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I.  INTRODUCTION 

For  the  past  ten  years  the  mathematical  techniques  of  optimization 
have  been  rapidly  improving  and  finding  wide  applications  in  various  fields. 
The  purpose  of  optimization  is  to  design  a  process  in  such  a  way  that  under 
certain  restrictions  the  maximum  profit  may  be  obtained.  Among  many 
techniques  for  solving  the  problems,  the  calculus  of  variations,  the  method 
of  gradients,  dynamic  programming,  and  the  maximum  principle  are  generally 
regarded  as  the  most  powerful  ones.  Each  of  these  methods  has  comparative 
advantages  over  one  another,  depending  on  the  problem  to  be  solved. 

Theories  of  the  calculus  of  variations  and  the  method  of  gradients  were 
well  and  fully  developed  long  before.  In  195?  Bellman  presented  the  method 
of  dynamic  programming  (1)  which  is  very  useful  in  the  solution  of  multistage 
decision  processes.  In  1956  Pontryagin  proposed  the  maximum  principle  (2) 
for  treating  the  time-optimal  continuous  processes.  The  first  attempt  to 
extend  it  to  the  optimization  of  stagewise  processes  was  made  by  Rozonoer  (3) 
in  1959*  Chang  (^)  and  Katz  (5)  independently  presented  the  discrete  version 
of  the  maximum  principle  in  I960.  The  generalization  of  the  discrete  version 
to  treat  different  complex  processes  was  made  by  Fan  and  Wang  (6)  and  Aris 
and  Denn  (7)  recently. 

Every  chemical  plant  consists  of  many  interconnected  process  units,  each 
of  which  is  either  discrete,  continuous,  or  a  combination  of  both.  The  discrete 
processes  can  generally  be  described  by  a  system  of  difference  equations 
and  the  continuous  processes  by  a  system  of  differential  equations.  A 
process  which  is  composed  of  both  discrete  and  continuous  processes  is  referred 
to  as  a  composite  process. 


As  a  process  becomes  complicated,  the  number  of  operating  variables 
increases  and  the  techniques  used  to  treat  the  problems  are  therefore 
restricted.  The  generalization  of  the  maximum  principle  to  treat  complex 
continuous  processes  has  been  presented  by  Fan  and  Wang  (8).  The  main 
purpose  of  the  present  thesis  is  to  extend  the  maximum  principle  to  treat 
topological!/  complex  composite  processes  and  derive  x*orking  algorithms 
that  overcome  the  difficulties  which  the  other  methods  may  encounter. 

The  underlying  idea  of  the  maximum  principle  is  to  transform  the 
original  system  of  equations  to  a  new  system  of  equivalent  equations  such 
that  the  new  system  is  easier  to  handle  and  clearer  to  visualize.  The 
relationships  between  this  transformation,  the  calculus  of  variations,  and 
dynamic  programming  are  also  discussed  in  detail. 


II.  THE  MAXIMUM  PRINCIPLE 

The  continuous  maximum  principle  was  first  developed  by  the  Russian 
mathematician  Pontryagin.  His  original  version  was  essentially  used  to 
treat  control  problems.  In  this  chapter  the  procedures  for  solving  problems 
and  the  basic  algorithms  for  simple  continuous  and  discrete  processes  are 
derived. 

Algorithms  for  Simple  Continuous  Processes 
A  simple  continuous  process,  shown  schematically  in  Figure  1,  can  be 
described  by  differential  equations  of  the  form 
dx. 

%(t)  =  g^i-  f± (^(t),  x2(t) xs(t);  e1(t),  e2(t) er(t)) 

tQ  <  t  <  T  ,     i  =  1,  2 s 

x  (t  )  =  9.   ,     i  =  1,  2,...,  s 


or,  in  vector  form, 


x(t)  =  g  =  f(x(t);  8(t))  ,     tQ  <  t  <  T 
x(tQ)  =  or 


(1) 


where  x(t)  is  an  s-dimensional  vector  function  representing  the  state  of  the 
process  at  time  t,  0(t)  is  an  r-dimensional  vector  function  representing 
the  decision  (or  control)  at  time  t,  and  or,  a  constant  vector,  is  the  initial 
value  of  the  state  vector  x(t). 

A  typical  optimization  problem  associated  with  such  a  process  is  to 
find  a  piecewise  continuous  decision  vector  function,  9(t),  subject  to  the 
constraints 

♦  [e^t).  e2(t)....,  er(t)j<o  ,    i  =  i,  2,...,m     (2) 

which  makes  a  function  of  the  final  values  of  the  state 
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S=  S  cx.(T)  ,        c.  =  constant  (3) 

an  extremum  when  the  initial  condition  x(tQ)  -  a  is  given.  Ihe  function 

S  =  S  c^x^T),  which  is  to  be  maximized  (or  minimized),  is  called  the 
i=l  •L-L 

objective  function  of  the  process.  The  decision  vector  so  chosen  is  called 
an  optimal  decision  vector  function  or  simply  an  optimal  decision  and  is 
denoted  by  §(t). 

The  procedure  for  solving  the  problem  is  to  introduce  an  s-dimensional 
adjoint  vector  z(t)  and  a  Hamiltonian  function  H  which  satisfy  the  following 
relations: 

H(z(t),  x(t),  e(t))  =  2  z±£Ax(t);   8(t)).  (4) 

i=l 

dZ.  ^TT 

—ia  -  f£-  ,        1-.1,  2,...,  s.  (5) 

dt     ox^ 

z±(T)  ■  0±  t  i"  li  ••••••  s.  (6) 

It  will  be  shown  that  if  the  optimal  decision  vector  8(t)  is  interior  to 
the  set  of  admissible  decisions  8(t),  given  by  Equation  (2),  a  necessary 
condition  for  S  to  be  an  extremum  with  respect  to  0(t)  is 

ff.O.  (7) 

If  9(t)  is  constrained,  the  optimal  decision  is  determined  either  by  solving 
Equation  (?)  for  9(t)  or  by  searching  the  boundary  of  the  set. 

Assume  that  the  function  f(x(t);  6(t))  is  continuous  in  its  arguments 
and  the  first  partial  derivatives  exist  and  are  piecewise  continuous  in  their 
arguments.  Let  x(t)  be  the  optimal  state  vector  of  the  process  corresponding 
to  the  optimal  decision  6(t),  then 


|f=f(x(t);  G(t)). 

6 
(8) 

A  small  perturbation  of  9(t)  from  §(t)  can  be  represented  by 

G(t,e)  =  0(t)  +  ecp(t)  +  0(e2), 

(9) 

and  the  resulting  perturbation  of.  x(t)  is  then 

x(t,e)  =  x(t)  +  ey(t)  +  0(e2). 

(10) 

•where  e  is  a  small  number,  cp  and  y  are  bounded  functions  of  t,  and  0(e  ) 

?                                  2 

denotes  the  e  term  and  those  of  order  higher  than  e  . 

A  variational  equation  can  be  obtained  from  Equations  (1)  and  (8)  as 

dy .             « 

e  — i  =  f,(x;6)  -  f,  (x;9)  +  O(e^)  ,    i  =  1,  2 s. 

dt    l       x 

(11) 

Expanding  Equation  (11)  in  a  Taylor  series  around  (x(t);§(t)),  one  obtains 

dy.    s     df.(x;6)    r     of. (x;G)     ? 
e  _i  =  2  cy  -| +  E  ecp.  -i- +  O(e^)  , 

(12) 

i  =  -L,  <~,  •  •  •  |  s 

where  the  partial  derivatives  are  evaluated  at  (x(t);6(t)). 

If  Equations  (5)  and  (12)  are  substituted  into  the  expression 

s         s     dy±    s     dz± 
cE  j^  ^izi  ■  Jx  ezi  dT  +  .^  e^i  dT 

(13) 

one  obtains 

,,,   s         s      s     of.    r     3f.     9 
4  2  ev  z<  =  2  z.  ,  2  ey.  — i  +  2  «?4^  +  0(O 
^  i=l   *  *   i=l  x  Lj=l   J  Sxj   j=l   J  S9j       J 

+  2  ey  (-  ||-) 
i=l   1   dxi 

s   r    df 
=  2   2  z.  ^i  ecp  +  O(e^). 
i=l  j=l  x  "j   J 

(14) 

Considering  the  linear  terms  in  Equation  (14)  and  integrating  from  t 

=  t0 

to  t  =  T  yields 

Since  the  initial  value  of  x(t)  is  given  and  fixed,  we  have 

eyi(tQ)  =  0  ,        i  =  1,  2 s.  (16) 

Substitution  of  Equations  (6)  and  (16)  into  Equation  (15)  and  use  of 

the  definition  of  the  Hamiltonian  gives 

s  T  r 

£  ec,y,(T)  =  f  t    H-  e<p.  dt.  (17) 

i=l   X  X  0  j=l  dyj   J 

Suppose  the  objective  function  has  a  maximum.  The  quantity  on  the  left 
hand  side  of  Equation  (17)  is  the  variation  of  the  objective  function  S, 
which  must  be  zero  along  the  optimal  trajectory  for  free  variations  (un- 
constrained variation)  and  be  negative  for  variations  at  the  boundary  of 
the  constraints,  that  is, 

s 

S  ec.y.(T)  <  0  ,  (18) 

i=l 

Thus  from  Equations  (17)  and  (18)  we  conclude  that  for  arbitrary  ecp 
the  necessary  conditions  for  S  to  be  a  maximum  are 

§g--0  at  6j(t)  »  GjCt)   ,       tQ<t<T,       j  =  l,  2 r.     (19) 

J 

when  8  .(t)  lies  in  the  interior  of  the  region  of  6(t),  or 

H  =  max   at  ©,(t)  »  e\(t)  ,   tQ  <  t  <  T  ,   j  =  1,  2 r  (20) 

when  0j(t)  lies  at  the  boundary  of  the  constraints.  If  the  objective  function 
S  has  a  minimum,  then  reversing  the  inequality  signs  in  the  above  derivations 
will  give  H  =  min.  in  Equation  (20). 
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Algorithms  for  Simple  Discrete  Processes 

A  simple  discrete  process,  shown  schematically  in  Figure  2,  can  be 

described  by  difference  equations  of  the  form 

n   _n   n-1   n-1      n-1   n   n      n 
x^  =  T  (x]_  ,  x2  »  •••»  xs   »  9l'  62*  ****  er^'   i  =  1,  2,  ...,  s 

0 
.   *i  =  v 

or,  in  vector  form, 

n   „n  /  n-1  .n> 
x  =  x  (x   ;  9  ). 


(21) 


x°=o-. 


where  x31  is  an  s-dimensional  vector  function  representing  the  state  of  the 
process  at  the  nth  stage,  8n  is  an  r-dimensional  vector  function  representing 
the  decision  (or  control)  at  the  nth  stage,  and  or  is  a  given  constant  vector. 
The  adjoint  vectors  and  the  Hamiltonian  function  are  defined  as 

s 
i=l 

n-1  „n, 


Hn(zn,  x",  en)  =    £    z5T?(xn-1;  0n)   ,  n  =  1,  2,   ...,  N.  (22) 


n-1        SB0  ?      n  ^Lj£l  i  =  1'2*   '-■ 

1  ox?"1      0s1    3        ox?'1  n  =  l,  2,   ...,  N. 


N 
2^=0^,  i  =  1,  2,   ...,  s.  (2ij.) 

The  objective  function  is  generally  of  the  form 

S  «  E  c4xf  .  (25) 

i=l 

The  necessary  conditions  for  S  to  be  an  extreraum  are  found  from 

^S_  =  0  ,    n  =  1,  2 N.  (26) 
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or  by  searching  the  boundary  of  the  set  of  0(t). 

The  proof  is  parallel  to  that  of  the  continuous  case.  Let  6n  be  the 
optimal  decision  and  5?1  the  corresponding  state  vector.  Then  we  have 

en  =  en  +  €9n  +  o(e2)  ,    n-1,2 n.         (27) 

x*1  m  x"  +  ey*  +  0(c2)  ,    n  =  1,  2 N.  (28) 

Combining  Equation  (21)  and  (28)  gives 


cyn  =  tW)  -  fff^ll1)   +  0(e2)  ,  t     x 

(29) 
i  =  1,  2,  ...,  s;   n  =  1,  2,  ...,  H. 

Expanding  Equation  (29)  in  a  Taylor  series  around 

i  =  1,  2,  ...,  s;  n  =  1,  2 N 

where  the  partial  derivatives  are  evaluated  at  (aP*1;?*).  Multiplying 
Equation  (30)  by  2?,  summing  on  n  from  n  =  1  to  n  =  N  and  on  i  from  i  =  1 
to  i  =  s,  and  employing  Equations  (23)  and  (22)  yield 

3       r  v  «       ft  in       N      s      r      „  oTV(xn~  ;9  )  2 

2    e  I  yM  -  AJJ  =2      SI    J  -^-3 «£  +  ©CO-     (3D 

j^L      1*1  i      'I  U      n-i  i=i  j=i    1  ^n  3 

Since  the  initial  value  of  x11  is  given  and  fixed,  we  have 


eyi  =  0  ,    i  =  1,  2,  ...,  s.  (32) 

Considering  the  linear  terms  in  Equation  (31)  and  using  the  definition  of 

the  Hamiltonian  and  Equations  (24)  and  (32),  one  obtains 

s        N   r  __n 

2  ec.y?  =  S   S  |f  «p»  .  (33) 

i=l   *  i   n=l  j=l  aej   J 


11 


Suppose  that  S  has  a  maximum.  The  left  hand  side  of  Equation  (33)  is  the 
variation  of  the  objective  function  and  must  be  zero  for  free  variations 
and  be  negative  for  variations  at  the  boundary  of  the  set  of  Qn,  that  is 

2  ec.y?  <  0  .  (3*0 

Thus  from  Equations  (33)  and  (34)  we  conclude  that  for  arbitrary  €cpn 
the  necessary  conditions  for  S  to  be  an  extremum  are 


>n 


aH  =  0    at    6n  =  6n  .   n  =  1,  2 N 


ae* 

when  6"  lies  in  the  interior  of  the  region  of  6",  or 

H  =  max    at    9n  =  Gn  ,   n  =  1,  2 N  (36) 

when  Gn  lies  at  the  boundary  of  the  constraints. 

It  is  to  be  noted  that  the  necessary  conditions  for  S  to  be  a  maximum 
(or  minimum)  in  the  continuous  case  can  be  strengthened  so  that  if  S  is 
maximum  (or  minimum),  then  H  is  a  maximum  (or  minimum).  However,  in  the 
discrete  case  there  is  no  such  analog.  The  strengthened  condition  will 
be  proved  later. 
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HI.  OPTIMIZATION  OF  COMPOSITE  PROCESSES  AND  APPLICATIONS 

The  optimization  of  complex  discrete  processes  and  complex  continuous 
processes  has  already  been  treated  extensively  (6,  8).  Many  processes  in 
practice,  however,  are  encountered  in  the  combined  form.  For  convenience, 
such  a  complex  process  is  called  a  composite  process.  In  this  section  a 
general  method  of  obtaining  directly  the  optimal  policy  for  a  composite 
process,  without  decomposing  it  into  subprocesses,  is  presented. 

Statement  and  Derivation  of  the  Algorithm 

i 

The  algorithm  is  applicable  to  a  composite  process  consisting  of  three 
basic  types  as  shown  in  Figures  3»  ^»  and  5» 

In  a  simple  continuous  process  contained  in  a  composite  process,  the 
change  of  the  state  can  be  described  by  the  following  performance  equations 

abtt 

~^=fi(x1(t),  XgCt),  ....  xs(t);  ex(t) er(t)),  tQ<t<T 

X  =  J.  i  ^-j  •••»  s 
or,  in  vector  form, 

g=f(x(t);  e(t))  .  (1) 

For  a  simple  discrete  process  contained  in  a  composite  process  the 
performance  equations  are 

Xj  =  i  ^Xt   ,  Xp   ,  •••»  X    S  v-j,  Op,  •••»  ■  _/  ,    1—  X,  £,•••,  S 

or,  in  vector  form, 

xn  =  ^(x"'1;  en).  (2) 

All  of  the  composite  processes  are  composed  of  several  inter-connected 
branches.  The  point  where  two  or  more  branches  connect  is  called  a  junction 


dl)(t) 
*     i     1    J     1 


13 


e(2)(t) 
*     *     *    *     * 


31        32  3n  3N 

— -q— [2p---H — [If— 


*   i i * t 


(rw)l    (N-02      (N-I)n    e(N-ON 

-HZLl — dD — ED — Cil — -N-i 


i L 


£ 


to 


_L__ 


*N 


(a)     with     continuous     first     branch 

el2)(t) 

\      h       III 


0n      012      0ln 


-0 — QjQ-~ 0-- 


G 

i 


IN 


03.     032        03n         03N 


*         *        * t ^L 


h 


CN-Dl     (N-02      (N-!)n    ^(N-I)N 

e.      0       e 


Q— gj-— fn>  — 4n] -n-i 


1      I     I      i     I 


N 


(b)     with     discrete     first     branch 


Fig.  3.  Separating     points     in    a     composite    process. 
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point.  There  are,  as  shown  in  Figures  3»  ^  and  5»  three  basic  types  of 
junction  points  (6,  8): 

(a)  A  separating  point  is  a  point  where  one  branch  of  the  path  splits 
into  several  branches. 

(b)  A  combining  point  is  a  point  where  several  branches  combine  into 
one  single  branch. 

(c)  A  crossing  point  is  a  point  where  several  branches  combine  together 
and  then  split  again  into  several  branches. 

The  relationships  between  the  state  vectors  of  different  branches  at 
the  junction  points  are  described  by  the  following  junction  equations.  They 
are  as  follows: 

(1)  Separating  points 

a.  If  the  first  branch  (entering  stream)  is  continuous  (see  Figure 


3a) 


3b) 


X(k)(0)   or   xk0  =  g^(x^(T1)).   k«2.  ....  I.      (3)* 
b.  If  the  first  branch  (entering  stream)  is  discrete  (see  Figure 

x(k)(0)   or   x*°  =  g^k>(x1N)  ,    k  =  2 N.        (4) 

(2)     Combining  points  (see  Figure  b) 

x<K><0)   or   xN0  =  g«"(xU.  x2" x«,  x<d'+1>(Td,+1) 

x(N"1)(TN-l>)-  <5> 


*  The  nth  stage  in  the  kth  discrete  branch  is  denoted  kn,  and  the  kth 
continuous  branch  by  (k).  Tk  is  the  length  of  the  kth  continuous  branch. 
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c'  and  d'  denote  the  number  of  continuous  branches  and  the  number  of  discrete 
branches  entering  a  junction  point  respectively.  Similarly  c"  and  d"  denote 
the  number  of  continuous  and  discrete  branches  leaving  the  junction  point 
respectively.  The  total  number  of  the  continuous  branches  is  represented  by 
c  and  that  of  the  discrete  branches  by  d. 
(3)  Crossing  points  (see  Figure  5) 

X«(0)  or  xk0  =  g<k  V.  x2K x™,  *(d'+1)(Td,+1) 

*W(Tm))  .  <«) 

k  ■  m+1,  m+2,...,  N. 
The  optimization  problem  under  consideration  may  be  stated  as  follows: 
Given  all  the  performance  equations,  junction  equations,  lengths, 
number  of  stages  and  initial  states  of  a  composite  process  consisting  of  a 
path  with  N  branches,  b  initial  points,  and  e  final  points,  find  the  optimal 
decision  vector  function  of  each  continuous  branch  and  the  optimal  decision 
vector  at  each  stage  in  each  discrete  branch,  so  as  to  maximize  a  certain 
linear  function  of  the  final  states  of  the  process,  the  objective  function, 


2   !(Cn)  c(c")^(c»)^  wv   !(dn).d''iUi''N 
c"  i=l 


:!c"y5")(ln)+E   !    cd'V 
11    c    d"  i=l   1   x 


where  c.  are  constants;  the  superscripts  c"  and  d"  denote  the  labels  of  con- 
tinuous and  discrete  branches  respectively;  £(|  and  E,  indicate  the  summa- 
tion over  all  continuous  and  discrete  branches  with  end  points  respectively. 

Similarly,  S.  and  2  are  used  to  denote  the  summation  over  all  continuous  and 
c'    d' 

discrete  branches  xjith  initial  points  respectively  and  £  and  £  to  denote 

o    d 

the  summation  over  all  the  continuous  and  discrete  branches  respectively. 
To  find  the  optimal  decision  vectors,  adjoint  vectors  and  Hamiltonian 
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functions  satisfying  the  following  relationships  are  introduced  as  follows: 
(l)  For  each  of  the  continuous  branches 

oo 

H(k)(x.z.9)  =  £  zWfWOc*) x&g,,  e*) eWj.  (7) 

TF^"  -^TkT  '      i  lf  2* s         (8) 


(2)  For  each  of  the  discrete  branches 
s 


Hkn=  s  2f  Tf  Cx*^ x*^;  e*0,  e1^,...,  e^).   (9) 


^■dfer-         isl«2 s(k)-         (10) 

The  values  of  the  components  of  the  adjoint  vectors  at  the  final  points 
of  the  entering  streams  of  each  branch  satisfy  the  following  relations : 
(a)  Separating  points 

i)  If  the  first  branch  is  continuous 

e(c")  >fr (c»)  (d")  .  (d«) 

c"  j=l   ox^x;   J       d»  j=l   ox£i;   J    (11) 

i-1  2     s^ 
ii)  If  the  first  branch  is  discrete 

(c»)   (c")  (d»)  a  (d«) 

2i  -  S   2      IN   zi   (°)+s   s      ik   z-j  » 

c"  j=l        bxT         J       d«  j=l    axf    J        (12) 


i  =  1,  2,...,  s^. 

(b)  Combining  points 

For  the  case  in  which  branch  N  is  either  continuous  or  discrete 
one  has  the  following  relations: 
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i)  For  the  continuous  entering  branches 

(N)  -   (N)  1-1,  2,...,   s<k> 

***>-*  2HB>(oK    >->.« •••     3 

ii)     For  the  discrete  entering  branches 

(N)  »  (N)  4-19  .00 

...  SV        dg^  Aw\  1=  If    2,...,    s 

0=1      fccf       J  k  =  c'+l,  c'+2,...,  N-l. 

(c)     Crossing  points 

i)     For  the  continuous  entering  branches 

Co")  a  (c")  (d»)       (d»)  _  (k) 

s         og^         /„«>     .  s         agi         dHA      x  -  x,<s,...,z 

(Tk)  =22  -fcr-  z  C  ^0)  +  E      2  -4-r-  z*  °. 

k        c»  J=l        ox£k;       3  d»  j=l        ox£K;       j  k-1 c». 

(15) 


ii)    For  the  discrete  entering  branches 

(c»)  4  (c»)  (d»)  .   (d»)  .  -   ,  (k) 

v.M  S  dg.,  /_n\  S  dg*  JttA  1=1,2,...,    s 

zf  =  2      2  -%-  z  °  5(0)  +  2      2  ~^p-  z*  °,  (16) 

x        c"  J=l        cfltf        J  d"  j=l        &cf        J  k=c'+l m. 

(d)  Final  points 

(17) 
d"N    d»N  .  _  ,  ,     e(d") 

Zj    =  Cj     |  X  —  X,  £,...,  5      . 

The  optimal  decisions  for  every  continuous  branch  are  then  determined  from 
the  following  necessary  conditions: 


*  Equations  (13)  and  (14)  are  written  for  the  case  branch  N  is 

continuous.  If  it  is  discrete,  z^  '(0)  in  the  equations  is  replaced 
oy  z.  . 
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^j^r  =0   or   H^  =  maximum  at  every  point  t,  tQ  <  t  <  Tk    (18)* 
and  for  each  discrete  branch 


or 


1  \ty\ 

H      =  maximum  ,.    if  5    lies  on  the  boundary 
hrf™  _  A        ,*    *kn 


(19) 


ae1" 


■  0  ,  if  5   is  on  interior  point. 


Let  6"'k'(t)  and  0   be  the  optimal  decision  vector  functions  of  the 
kth  continuous  and  discrete  branch  respectively  and  £*  '(t)  and  x^  be 
the  corresponding  optimal  state  vector  functions  of  the  kth  branch.  Then 
we  have 

6(k)(t,e)  =  9(k)(t)  +  e<p(k)(t)  +  0(e2).  (20) 

ekn  =  §kn  +  e(pkn  +  Q(e2)#  (21) 

and 

X(k)(t,e)  =  x(k)(t)  +  ey(k)(t)  +  0(e2).  (22) 

x^1  =  ^  +  ey^  +  0(e2).  (23) 

By  means  of  a  Taylor  series  expansion,  the  variational  equations  are  obtained 
as 

^  (k)    (k)      M-OO^OO.stk),. 

6  ^t~  -  *   eyj     77k}    +  ri  u  *e  ; 
o=i        axj 

_f(k)(-(k);-(k))+0(e2)>  ^ 

for  each  continuous  branch,  and  as 

*  It  vail  be  proved  in  Section  IV  that  these  two  conditions  are 
equivalent  for  the  continuous  branches. 
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_Jcn/-k(n-l)  -knN 

kn   J   k(n-l)  5Ti  (x     ;6  )  .  Jav-k(n-l).flkn, 

eyi  =  A  eyJ     ^kl^l) +  \   (x     ;Q  > 

J=J.  ox. 

for  each  discrete  branch. 

The  relationships  between  the  variations  of  the  state  vectors  of 
different  branches  at  the  junction  points  are  obtained  by  expanding  the 
junction  equations  in  power  of  ey.  as  follows: 

(a)  Separating  points 

i)  If  the  first  branch  (entering  stream)  is  continuous 

s(D  -      9g(k) 
«y£°(0)  or  «yf-  I   ey^1)(T1)-^r7  +  0(e2),  (26), 

i-  1  2      s(k) 
k  =  2,  3,...,  N. 

ii)  If  the  first  branch  (entering  stream)  is  discrete 

S(D     Sg(k) 

ey|k)(0)  or  eyf  -  S    eyf  -^  +  0(e2),         (2?) 

0=1     ax.. 

k  ■  2,  3.»..»  N. 

(b)  Combining  points 

>')  ..  (H) 

*FW    or  eyf  -  S   I    „<<"><,  >  *1   + 

c'  j=l     J       dxi 

*  For  the  kth  continuous  branch,  the  first  term  of  the  left  hand  side 
is  considered,  and  for  the  kth  discrete  term,  the  second  term. 
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s 


9 

(d«)     a  (N) 


»N  Bgi  '  ,  n,   2N         ,   o      _(N) 


+  s  "s   «y<TM-4re  +  o(c2).   1-1.2 «w.  (28) 


i      •*  d'N 


(c)  Crossing  points 


.(d«)     a  (k) 

+  S   I     eyd'N   i   +0(e2),         (29) 

d»  j=i    J  axj 

L  -  1  2     s<k) 

k  =  m+1,... ,  N. 

Since  all  the  initial  states  of  the  process  are  given  and  fixed,  one  has 

/,  \      i/\  i  =  l,  2,».»,  s 

y(k)(0)=ykO  =  0t  (30) 

K  =  X,  c , • • • ,  D  • 

where  b  denotes  the  number  of  entering  branches  with  fixed  initial  states. 
For  the  continuous  branches,  consider  the  relation 

00  (k)     .  (k)    (k)      (k) 

Substituting  Equations  (8)  and  (2k)   into  (31)  yields 

.00         .<k) 
4  I     ey.^k)=l  ZiH4k)^(k)^(k))-4k)(x(k);e(k))J  +  o(e2). 

G.  U  ^  _->     XX      j  _-i    1   L  X  X  J 

x-1  x-1  (32) 

Integrating  Equation  (32)  from  t  =  0  to  t  =  T,,  one  obtains 

(k) 
e  Jx    |/ik)(Tk)  z^k)(Tk)  -  v<k>(0)  z<k>(0)J 

'     '  (33) 

T  a0O 
=  S    2      ■£O|f£>0ECk)ltW)  -  f(k>(x<k);5(k>)J  dt  +  0(e2). 
0  x=l 


23 


For  the  discrete  branches,  multiplying  Equation  (25)  by  z!?1  employing 

(k) 
Equation  (10),  and  summing  from  n  =  1  to  n  =  N  and  i  =  1  to  i  =  sv   give 

M 

£   ,  kN  kN   k0  kO. 


=  E   2   z. 
n=l  jrf.   x 


^^(n-D^knj  .  ^(n-D.^j  +  ^  §   ^ 

Summing  Equations  (33)  and  (3^)  over  all  the  continuous  and  discrete  branches 
respectively  and  adding  the  resulting  equations,  one  has 

« j^c)<Tc>  4c)<v  -  ^c)(o>  4c)co)j ♦«  jj%f  ■?  -  *f  «f] 

.(c) 


J    s 

r  s 

c  0    i=l 


s ;  2   4c)Lfio)^(a>?6<c))  -  4c)(*(c>;5(c))J dt 


N       s(d) 
+  2    2      2      zfl  lftflCa^)jedn)  -  rfo^dCn-D.gdnj  |  +  0<<2).         (35) 

d  n=l  i=l      i  L  x  1  J 

For  the  branches  containing  the  initial  points  or  states,  one  has  in  the 
left  hand  side  of  Equation  (35) 

«yp°(o)  z£k)(o)  =  o  , 

and  k  =  1,  2,...,  b        (36) 

kO  kO  n 
eyi  zi  =  0  . 

In  the  following,  the  three  basic  types  are  considered  separately. 
It  will  be  shown  that  they  lead  to  the  same  conclusion,  namely  Equations  (18) 
and  (19).  For  those  branches  connected  to  a  separating  point,  if  the  entering 
stream  is  continuous,  one  has  from  Equations  (11)  and  (26) 


*  S^'1)  indicates  the  optimal  value  of  xk^n"1\ 
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(o»)  Jd») 

c"  i=l  1  x  d"  i=l  1        1 

(c»)     (1)  .   (c-») 

S  S  /n\  dgj  /■    n\ 

c»  i=l      j=l        J        X    tar' 

J 

(d»)    0)  ^(d") 

s        s         /-,\         eg.         jiiq 

+  2  s     2   vfw -^fcr  4 

d«  i=l      o=l  tai  ' 

J 

(1)  (c»)       <o«)  (d»)       (d») 

s  (1)  s  °Si  (cn)  s  °si  d"0  i 

=    2      cy^^T  )  Is      S  -^jy-  4°  ;(0)  +22  -^y-  2J  °J 

o=i      3      1  Lc»  i=i      tar'     x  d«  i=i      axf'     x  J 

0  vl 

s(l)  s(l) 

=    2      ey^(T)  2^(T)=    s      V±    (ft)  ^'ftl)  •  (37) 

The  left  hand  side  of  Equation  (35)  becomes,  by  using  Equations  (36)  and  (3?): 

C»  i=l     L  1      CI      C       X         1       J    d„  i=1     L  X     J. 

(1) 

,(o")  s(d") 

.1  £   cy^>(T„)^">(T„)+r  r   *f».f.  (38) 

c"  i=l    x    ci    c    d„  i=1    1    1 
where  the  last  bracketed  quantity  on  the  left  hand  side  of  Equation  (38)  is 
separated  from  the  first  bracketed  quantity  on  the  left  hand  side  of  Equation 
(35).  Recall  that  d"  =  d  for  this  case.  Equation  (35)  thus  becomes 

(c»)  (d») 

r.   £     (c")/m  n  (o")/»  n  .-   I     d»N  d»N 
2   2    ey^  '(?  t)  z>  /(Te„)+2   2    ey.  z± 

c»  i=l    X  °        X         d»  i=l 
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(c) 

=  s  F  s  z.(c)i  f(chzic);Q(c))  -  f,(c)(x(c);e(c))  I  dt 

c  J0  ^  i  L  x  i        J 

(d) 

+  e  2  !  a?l!if^-a);edn)-afG?iCn-1);5dn)J  +  o(«2).   (39) 
dn=li=l   iLl  x  J 

If  the  objective  function  is  to  be  maximized,  the  perturbation  of  the 
decision  variables  can  only  be  to  make 

(c»)  (d») 

s      (r«)    (c»)  s      d"N  d"N  i  ,,  n\ 

S   2    ccT  Vi  ;(TC„)  +  2   2    ec"  yj  "  <  0  .     (40) 
Lc«  i=l    l   *    c    d»  i=l    1    x  J 

Combining  Equations  (17).  (39),  and  (40)  gives 

2  rT    2  °  z[°\  fCc)(2(c).e(c))  .  f(c)(X(c);e(c))j  dt 

(d) 

+  222       zfLTf1^^^)  -  if  (xd^1!edn)J  +  0(e2)  <  0.     (41) 


or 

2  rT    2      z(c)i  f^hx^^c))  -  f  <c>(x(c>;e(c>)  I  dt 

c  0     i=l      *    L  *  x  J 

N      s(1)  ,       r        ,     M^ix^-VlQ6*) 

+  222      z^i    2     ecp^  -i a I  +  0(e2)  <  0  .  (42) 

dn=li=l      lLj=l      3  agdn  J 

v 

Since  the  perturbation  of  each  decision  vector  is  independent  of  the 
perturbations  in  the  other  decision  vectors  both  in  the  continuous  branches 
and  in  the  stages  of  each  discrete  branch,  it  may  be  concluded  that  the 
integrand  of  each  integral  and  each  term  containing  a  set  of  independent 

variables  must  itself  be  non-positive.  Thus 

(c) 
2   z.(c)i  f,<c)(x(c>;e<c>)  -  Hc)(x(c>;e(c))  |  +  0(«2)  <  0  (       (43) 
i=l   l  L  1  x  J 
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and 

E  (ef-ef)  E   zf1-* s -  +  0(e2)<0,    W 

0=1    J       J    i=i    l         s§7 

which  are  equivalent  to  Equations  (18)  and  (19). 

For  those  branches  connected  to  a  separating  point,  if  the  entering 
stream  is  discrete,  one  has  by  using  Equations  (12)  and  (27) 
(c»)  (d'«) 

s       (rtt)  ((>"}  s      d"0   d"0 

E      2        tsrl0  ;(0)  4°  '(0)  +E      S        eyd°  z? 
c"  i=l  x  x  d"  i=l 

s(c»)  s(l)  a  (c«)  s(dM)s(l)       9e(d«) 

=  2      E*  |      e^!fi        z(o«)(0)+2      |        !    e^!fi      Zd"0 

c"  w.     j=i      J    W:N     x  d"  i=i   j=i     J   a»Hr   x 

0  3 


.    £.     <yU,E      £  Zi-_2(o»)(0)+r      S  ZL_  ■*■<>, 

J  Lc»  j=l  *JU      i  d»  i=l  .IN      i    J 


ax.  Sx. 

=    1      ey^  z^  =    E      ey1*  2M  .  (45) 

0=1        3       3        i=l        X      x 

5y  using  Equations  (36)  and  (45),  Equation  (35)  is  reduced  to 

(d»)  (c») 

E      I        eyf^+E      I        eyf  >(V)  z^^,,) 

d"  i=l  x        1  cw  i=l  x  c        x  c 

N      s(d) 
-E    E      E      zf^lf1  (^(n-iJje**)  -  lf1(xd(n-1);Gdn)J 
d  n=l  i=l 

T  s^ 
+  E  J    E      z^lf^x^e^)  -  f(c)(x(c);9(c))  I  dt  +  0(e2).     (46) 
c  0  i=l      l    L  *  *  J 

It  is  worth  recalling  that  c"  =  c  for  this  case. 

If  the  objective  function  is  to  be  maximized,  the  perturbation  of 

the  decision  variables  can  only  be  to  make 
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iS     S  *"  .cf N  yfN  +  Z      2°    .o^'Vi00^.)]  +  0(«2)  <  0.     (4?) 
Ld»  i=l         *       x         c»  i=l         X       *         °    J 


or 

N      g« 


222     8Jn[!i*1C3e,i(nJ-)sedn)  -  ^(S*^;^)]  +  0(c2)  <  0  , 
d  n=l  i=l 


(48) 


and 


(c) 
2  f    2      z(c)|  f(«>^o),0(o))  _  f(c)(2(c).§(c))  |  dt  +  0(e2)  <  0.       (49) 

c  0  i=l   i  L  i  1 

Following  a  similar  reasoning  as  before,  one  has 

(c) 

E  z.(c)|^c)(x(c);e(c))-4c)(x(c);e(c))j+o(e2)<o  ,      (50) 

and 

(d)     dn,-d(n-l)  -dn. 

•  I    ftf.lf)  I   2f   i(  ^     W)<o.         (5D 
j=i  3   3  i=i  x     se™ 

For  those  branches  connected  to  a  combining  point,  if  the  leaving 
branch  is  continuous,  one  has  from  Equations  (13).  (14).  and  (28) 
(N)  (N)     (c»)         a  (N) 

(d«)     a  (N) 

,.   I    d'N  5gi   i  (N),M 
+  2   2   ey.    d,N  \z±     (0) 

d«  j=l    J   ^j 
(N)  (C)  .  (N)  JlOJd')     a-(N) 

s  s         r„n    &*i   no      s  s   d'N  Sgi   (n),  x 

=  2   2   2    cyC,)(Tc,)-4^74N)(0)+2   2   2   ey*  -%%  zf;(0) 
c«  i^l  j=l    J    °   ox^C  '  X      d'  i=l  j=l    J   ox<j 

-«   5   eyf  >(T0.)  2   ^H»(fl«   I   eyf  2    ^J^W 
c'  j=l   J    e   i,l  3xjC  j        d«  j=l   J   i=l   axj 
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.<*')  s(d') 

=  2   2    ey^(Tt)Z|c,)(l.)+E   2    eyd'Nzd'N.         (52) 

Recall  that,  for  this  case,  d'  =  d.  The  left  hand  side  of  Equation  (35) 
becomes  by  using  Equations  (36)  and  (52) 
Co) 

=  »    WW  •L'^V  -  vtho)  4o)(o)j 


s 


(d) 


+  I    2       dyf  zf  -  yf  ■*  I 

diPl      Ll      x  1      1  J 

Ch)  (c») 

1=1  C1    1=1 

S(N)  8<c») 

-2      eyW(O)  z.(N)(0)  -Z      2        ey.(c,)(0)  8Cc»)(0) 
i=l        1  1  c«  i=l  x  i 

(d«) 

S(N) 
-    Z       ^N)(TN)  Zf^(TN)   .  (53) 

Equating  this  to  the  right  hand  side  of  Equation  (35)  gives 

(N) 

E      ey(N)(T  )  z(N)(T  ) 
K     eyi     UN;  2i     UN; 


i=l 


T     s^ 

2  r    2     z.(c>jf.(c)(*(c)J8(c))  -  f.Cc)cs(c);eCe))'|  dt 

cOi=llLl  x  J 


(d) 
N      s 

+ 


If  the  objective  function  is  to  be  maximized,  the  perturbation  of  the 
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decision  variables  can  only  be  to  make 

(N) 

«y«  <  o  ,  (55) 


2  ec:  y± 

i=l  x       J- 


or 


(c) 

S   z(c)l  f (c)(x(c);6<c))  -  f  ?c)(x<c);e"(c>)  I  +  0(e2)  <  0    (56) 
i=l   i  L  i  X  J 

I    (e^  -  S*1)  2  z*1  i(     '    ♦  o(c2)  <  0  .    (57) 
j=i  J   J  i=i  x     oe™ 

If  the  leaving  stream  is  discrete,  by  using  Equations  (13).  (1^).  and 
(28)  one  has 

(N) 
■    NO  NO 

&     6yi  Zi 


ML 

(N)     CcO  -,  (N)        (d«)     .  (N) 

8   I  S   •   (o«),   x  **  v   I  d»NSgi   I  N 

4.1  L«t  «   J   c  axr  ;  d>  i=i  J  taS WJ 


s   s     (c«),   x  Sgi    NO   „   5.   !     d'N  dgi    NO 

=  2   2   2    ey   ;(Tc,)-7^t4  +2   2   2  e7j  —^    z± 

c'  i=l  J=l  J  °       hxf  }     X        d«  i=l  0=1  J       9x5 

(c«)  (N)  4   (N)  Jd<)  (N).   (N) 

S  CcO,,,     x     *        **i  80Ar      ?  d'N^      !fi        NO 

=  2      2        ey^  ;(TC,)     2        ~ ?^7  zi    +S      2        e^j      Z      ^N  zi 

c"  j=l  3  °      i=l      &x      ;     X        d»  j=l  J     i=l    6x. 


>')  t  s(d») 

=  2      2        ey(.C,)(Tcf)  z^C,)(Tc,)  +2      2        eyV 


Note  that,  for  this  case,  c'  =  c.  By  using  Equations  (36)  and  (58),  the 
left  hand  side  of  Equation  (35)  becomes 

C  1=1  O  i— X 
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S(C)  ,  %        S(N) 

=  2   2    cy.^)(T  ,)  z!c')(T  ,)  +  2   <  «f 
c,  i=1    1    e»   i    c«   i=1   i   i 

s(d«)  _0O 

,  „   J     d»8  d'N    ^    NO  NO 
+  S   2    ey,   z^   -2   ey.  z± 

d'  i=l  i=l 

S(C)  S(N)  s(d') 

c»  i=l   x    c  x  c    i=l   x  x    d'  i=l    *■   x 

(c«)  (d») 

s     (c')/   n  (<*'),   n   „   ^     d'N  d'N 

c'  j=l    3        3  d»  j=l    J    J 

s<N> 
=  2   cy1  «NN  ,  (59) 

i=l   i   i 
Equating  this  to  the  right  hand  side  of  Equation  (35)  gives 

(N)  -   (c) 

i    sy™  *?  =  2  r   2  z(c)|  f<c)(x(c);e(c))  -  f <c)(x(c);e(c))J  dt 
JwL   X   X    c  '0  i=l    L  1 

+  2  2   2  z^j^^^^e^)  -T?a(xd^-l);6d^)|  +  0(e2).  (60) 
d  n=l  i=l  1  L  x  i  J 

Therefore 

2  °  4C)|  4C)(X(C);9(C))  -  4c)(x(c);e(c))j  +  0(e2)  <  0  ,      (61) 
i=l   X  L 

and 

r(d)  s(d)    ^^(n-D.gdn) 

2   (0^-5^)  2   zf—i +  0(e2)<0.       (62) 

j=i   J   J  i=i  i     ae?1 

j 

which  are  equivalent  to  Equations  (18)  and  (19 )• 

For  those  branches  connected  to  a  crossing  point,  by  using  Equations 

(15),  (16)  and  (29),  and  recalling  that  2,  sums  over  all  the  entering 

c' 
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streams  that  are  continuous,  2 

d' 

suras  over  all  the  entering  streams  which 

are  discrete  and  2    and  2    sum  over  all  the  continuous  and  discrete  leaving 
cn          dw 

streams  respectively,  one  has 

(c«)                                                  (d«) 

2      2        cyf>(0)z£C,,)C0)+2      2        eyf  °  zf  ° 
c"  i=l                                           d»  i=l 

=  2      2        12      2       eyC') 
c»  i=l      Lc'  o=l        J 

a  (c)          s(d«)       a  (d«) 

(Ti)!fi         +2      1       .yd'H  dgi        |z(c«)(0) 
J                                          3 

+  2      2        J2      2       ey<c 
d»  i=l      Lc«  o=l        3 

a_(d«)           s(d«)        a  (d») 

s(c«) 

=  2      2        ey(cl>(T  ,)  z{c 
e«  j=l          °          c         3 

'>(T,)+2      2        eyd'Nzd'N 
C         d,  ^           0         J 

s(c») 

=  2      2      ey^C,)(Tct)  z^C' 
c'  J=l        J                    J 

(d>) 

)/m     \  ..  ^      1          d'N    d'N                         ,,^ 
(Tc,)  +2      2       ey,       z,       .                     (63) 

C          d»  j=l        °        3 

By  using  Equations  (36)  and  (63)  the  left  hand  side  of  Equation  (35)  can 

be  written  as 

(c) 
I  jl     eL^C)(Tc)  Z^C)(TC) 

-7iC)(0)zic)(0)J 

s(d) 
+  2    2      ciy^z^ 

-  yd0  zd0  I 
1      1  -i 

(c«) 

=  2      2      ey^'>(Tc,)  z£ 

c«  i=l        x          c        1 

c"  i=l                      CI           C 

e(d«)                                 e(d») 

+  2      2        eyd"KzfN+2      £       V'N«*"H 
d'  3=1          11          dtt  i=1          1        1 
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s(c»)  s(d") 

-IS      I         ^Cc-)(0)jB(c-)(0)+s      J         e^"02d»0| 
Lc»  i=l  x  x  d"  i=l  *       *    J 

(c«)  (d») 

s  (r»)  (cn)  s  d"N     d"N 

=  22         eyf  ;(T„)  4     ;(T-„)  +2      2         ey    %    "   ,  (ft) 

It  is  worth  recalling  that 

c  =  c1  +  c" 

d  =  d«  +  d" 

N  =  c  +  d 
Equating  the  right  hand  side  of  Equation  (6k)   and  the  right  hand  side  of 
Equation  (35)  gives 

(c»)  (d») 

2   2   ey<°  ;(TC„)  z>°  ;(TC„)  +2   2   ey*   z* 
c«  i^l    x    c    i    c    d„  i=1    1    1 

T  s^ 
=  2  r  2   z.(c)|  ffc)(x<c>;e(c>)  -  6chz(c)lQ(c))  I  dt 
c  0  i=l   X  L  X  J 

h   <d> 
+  2  2   2  2f  I  Tf  (x^1^^)  -  Tf  (5d(n-1);edn)j  +  0(e2).  (65) 
d  n=l  i=l  x  L  x  x  J 

Similarly,  if  the  objective  function  is  to  be  maximized,  the  perturbation 
of  the  decision  variables  can  only  be  to  make 

(d») 

1  d»N  d"N  .  n  ,,,, 

2  ecj   y.   <  0  .  (66) 
i=l    i   i   - 

and 

s(c») 

2   ecfc"V(cn)(T  „)  <  0  ,  (67) 

i=l    x   x    c 

or 
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s<c> 

E      z,(c)|  r>c>C5<c>j0<e>)  -  f  .^c>(3c(c);e^c>)  I  +  0(e2)  <  0  ,       (68) 


and 


2     (G^1  -  5*1)    2      z^  ~i 

>d      3  °       i=JL      1  39 


s    (e^1  -  e^1)   s     z*1  -i +  o(e2)  <  o  .        (69) 


Equations  (68)   and  (69)  are  equivalent  to  Equations  (18)  and  (19), 

Applications 

1.   A  simple  composite  process.  For  illustrating  the  use  of  the  algo- 
rithm for  composite  processes,  let  us  consider  the  problem  which  has  been 
solved  by  using  a  sequential  union  of  the  maximum  principle  and  dynamic 
programming  (9).  Ihe  composite  process  consists  of  a  discrete  unit  and  a 
continuous  unit  as  shown  in  Figure  6.  For  the  discrete  unit 

(70) 
10 
*1  =  Y  • 

For  the  continuous  unit 

a  (2) 

^L  (2)    LO(2) 

"dT~=  "  ^     +e 

(71) 
^2)(0)  =  xf. 

It  is  desired  to  minimize  the  total  cost 
P   +  P 


where 


(1)    11   rtll 
Pw  =  xx    =  n9   ,  (72) 


n  _ 


CO 
o 
o 
o 

>_ 
Q. 


C\J 


c 
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CD 

CO 
O 
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CD 
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T 

p(2)  =  lbr2L(xi2))2  +  (ei2))2Jdt- 

(73) 

To  solve  this  problem,  an  additional  state  variable  is 

Introduced  such  that 

< 

11    11  '11 

Xg  =  x-|_  -  u-9 

and 

xf)(t)  „  |  P*  [fe<2>)2  +  (e(2)}2j  dt  +  £L  m 

(74) 

This 

gives 

(75) 

and 

4zho)  =  £. 

(76) 

Hence,  for  the  continuous  branch,  one  has 

h<2>  =  42)  l-  4Z)  +  e(2>j  ♦  42)  [I  c*12)>2  +  § 

(e(2>)3 

» 

(77) 

dzl   ..  .  hH<fl  .   (2)  .  (2)  (2)      (2),  j  , 
dt      "  (2)   azl    z2   n  '     T.  u2; 
1 

•  o  , 

(78) 

Sx2 

(79) 

It  follows  from  Equation  (79)  that 

42)(t)=.l. 

Assuming  that  the  Kamiltonian  attains  its  minimum  at  an 

interior 

point 

of  the  region  of  6(t),  we  obtain  the  optimal  decision  by  setting 

an(2)  .  o 

• 
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■which  gives 

e(2)(t)  =  -  z(t)  =  e(t)  . 

(80) 

The  corresponding  x(t)  and  2(t)  are  found 

from  Equations  (71)  and  (78) 

as 

(2),..    11  1   Xt  .  .  -Xt  i 
»J  (t)  =  xx  [ore   +  ge   J  , 

(81) 

42)(t)  =  xf-  l<x(\+a)eXt   -  g(X-a 

\  -xti 

)e   J  , 

(82) 

•where 

X  =  J  a2  +  1  . 

»,g  are  constants  which  can  be  determined  from  the  boundary  conditions 

f.&rn,. 

and 

z[Z )(T2)  =  0  . 

Ihe  junction  equations  are 

*<2)<o)  =  £  . 

xfco)^. 

Hence,  by  using  Equation  (14),  one  obtains 

£  -  *<2)<o)  . 

11    (2)/Ax   , 

22   =  2^   (0)  =  1  . 

(83) 

Eie  Hamiltonian  for  the  discrete  unit  is 

JLl    11  >  10  .  .11  1  J  11  |  11 

H"  =  2X  L»l  +  e  J  +  22  Lxi 

-  »*] 

(2),M  I  10  ,  .11]  ,  ,  10 

♦  e11  -  ^)  . 

(84) 

Assuming  that  1T~^  is  stationary  in  the  interior  of  the  admissible  range  of 
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6  ,  one  obtains  the  optimal  decision  from  the  necessary  condition 

■which  gives 

z£2)(0)  =  -  (1  -  |0  .  (85) 

From  Equation  (82)  one  has 

^(0)  «  a*1  [«<JL4»)  -  p(X-a)j 

=  [x*°  +  enJ  |a(X+a)  -  P(X-a)]  .  (86) 

Solving  for  8   from  Equations  (85)  and  (86)  gives 

,11  _       -  q-u)       m  10 

B    o/(X+a)  -  3(X-aJ   *1 

=  ,(t^l)-v  •  W) 

The  minimum  cost  is  obtained  by  substituting  Equations  (80),  (81),  and  (87) 
into  Equations  (72)  and  (73)  and  adding  the  resulting  values  as 


where 


1  j    eXT2   e"^2 

c  p  2  L e  xt  ~  e JEEJ  "  consta^t' 

(XH-a)e  2  +  (X-a)e 


2.   Reactor  system  with  recycle  to  an  internal  point.  A  first  order 
consecutive  chemical  reaction 

kl       k2 

is  carried  out  in  the  reactor  system  as  shown  in  Figure  7»  The  initial 
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concentrations  of  the  reactants,  A  and  B 

are  y-i 

and  Yp  respectively. 

it 

is  desired  to  obtain  the  maximum  yield  of  the  intermediate  product  at  the 

outlet  of  the  system. 

To  illustrate  the  use  of  the  working 

alforithms  for  composite  processes, 

the  optimal  policy  is  obtained  only  in  the  form  of  differential  equations. 

For  the  first  branch 

40-Vx 

(88) 

k20e 

E2 

x1^ 

Y2   (89) 

where  t_  is  the  residence  time  and  is  assumed  to  be  constant. 

For  the 

second  branch 

(°) 

(90) 

dt      T.06      *1   • 

**z     _        '  r6<2>  (2)  m        " 

E2 
R8<2> 

x(2) 
x2 

(91) 

dt   "  K10e       *X    ^20® 

For  the  third  and  fourth  branch 

at   -°     •      *  "0- 

(92) 

4»              *<*> 

■  —  o           &  i   n 

(93) 

dt     °     '      dt   ~  °  ' 

The  junction  equations  are 

*[2)(0)=axf  +  3xf)(T4)  . 

x|2)(o)  =  oXf +  p44)(V  . 

(94) 

40 


The  adjoint  functions  are 


4» 

oH<2> 

(2)  ,     R9<2> 
21   k10e 

(2)  ,     R9<2) 

dt   ~ 

■  z2   k10e 

(95) 


(96) 


where 

q+r         '  pq+r 

x^3)(0)=x[3)(t3)=x|2)(T2) 

43)(0)=43)(T3)  =  42)(T2) 

^)(0)=Xf)(T^)  =  x{2)(T2) 

^(0)  =  ^(V  =  42)(T2). 
Then  the  objective  function  is 

S  =  zp)(T3).  (97) 

The  Hamiltonian  for  each  branch  is 

_L1      11    11   .     11    11  ,nQ, 

h     =x1    z1    +  z2    x%     ,  (98) 

„(2)„     (2)  %  (2)  ^2 

H        "  zl     .    dt       +  z2      TF^ 

%  % 

=  42)(-k10e"  "P:  42h  +  Zf)(k10e'  "^  x<2) 

T2j 


"  k20e     R9         ^   '  (99) 


H(3)  =  H(*0  _  0  u  ^100j 


dx| 
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h 


.  R9(2)  (   (2)    (2),  /imx 


.(2) 

&4* 


^2      3H(2)    (2)      R6<2>  ,   . 

«_  =  _  ^^  =  Z2   k2()e       f  (102) 


(3)  ,.(3)  ,»  ,.» 

=  o  .  (103) 


dz^;   ds£"   dzf"   dz2 


dt   "  dt   ~  dt     dt 
According  to  Equations  (13),  (14)  and  Equation  (94),  we  have  at  the  combining 
point 

zf  =  az;[2)(0)  ,  (104) 

z2JL  =  czf)(0)  ,  (105) 

.  zf  >(V  =  Pzf  \o)     ,  (106) 

zf  >%)   =  Pz^2)(0)  .  (107) 

Using  Equation  (11)  and  Equations  (95)   and  (96),  one  has  at  the  separating 
point 

^2)(T2)  =  zp^O)  +zf)(0)  ,  (108) 


z 


42)(t2>  =  43)<°>  +  4k)w  -  <io9) 

Since  the  objective  function  is  xip'(T ~) 

zp)(T3)  =  cp)  =1  ,  (110) 

•.  zp)(T3)  =  cp)  -  0  .  (Ill) 

It  is  seen  from  Equation  (103) 


42 

(k)              k  = 
z>  '  =  constant  , 

1               i  « 

•  3,  4 
=  1,  2  . 

(112) 

Combining  Equations 

(108),  (109), 

(110),  (111), 

and  (112)  gives 

■ 

42,c*2>  ■ 

■  4*>(D)  . 

(113) 

4*  )(,2) . 

=  l+zf5(0). 

(114) 

Combining  Equations  (104),  (105),  ' 

(106),  (107), 

(112),  (113),  and  (114)  gives 

zf  <T2)  . 

■  ^'(o)  =  4*: 

'(V  =  ^2)i 

:o) , 

(115) 

42)<v  ■ 

.  i  ♦  4«(o)  - 

l  +  2f}(V 

=  1  +  3z|2)(0)  , 

(116) 

and 

4%) 

-  Pzx  (0)  =  - 

ll 

zl  » 

(117) 

42,<V 

=  l  +  8z|2)(0) 

=  1+az2  ' 

> 

(118) 

Equations  (115),  (116).  (117),  and 

(118)  are  the  boundary  conditions. 

Assuming 

that  the  maximum  values  of  H   and  H^  ' 

occur  at  the  stationary 

points, 

one  obtains  the  optimal  5   and  §    from  the  conditions 

(119) 

and 

*h(2)  .  0 
5e<2>  " 

• 

(120) 

Solving  Equation  (120)  gives 

B»). 

R 

» 

n  =  kloE]L 

k20E2  ' 

(121) 

In 

4  ; 

-  z<2>> 

'2 
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Since  it  is  difficult  to  solve  Equation  (119)  for  6   explicitly,  one  shall 


define 


1     oxj1  x   OXJ1  r 


where  h-,  and  h2  are  the  implicit  forms  of  Equations  (83)  and  (89) 

h^xf,  xf ,  e22)  =  0  ,  (124) 

hz(*l'   ^»  811)  =  °  •  ^5) 

Differentiating  Equations  (124)  and  (125)  with  respect  to  9   gives 

ohn      ah.    ax}1     ah.    ax11 

_JL  +  -2-  ZL.  +  -1 — <L  u  0  (126) 

se11     ax^ae11     <**<»* 


ah2      ah2  ax^1    ah2   a**1 
a^  +  a^  a^  +  ixp  a^1 


2  +  _jl _i_  +  _2  _*  =  0  .  (127) 


From  Equation  (119)  one  has 

an11  n   11  **L  .  11  3x2  n28s 

Substituting  Equations  (122)  and  (123)  into  Equation  (128)  and  making  use 

of  Equations  (126)  and  (127),  one  obtains 

11         ah,       ahrt 

aii     a   -11   1  ,  ~U  2    n  ,,OQv 

Solving  Equation  (129)  for  8  yields 


m 


en  = S .  (130) 

11  11   11 

*1   z2  "  zl   1 

M*  -n (— ir-'J 


*2     22 


Since 


The  expression  for  the  maximum  yield  is  obtained  by  substituting  Equation 
(121)  into  Equation  (91) 


uhere 


*42) 

dt 

=  k1Q  In 

(2) 

h^2)^ 
X2 

42M2\fi 

x(2) 

xl 

-k2Qln 

x<2)     z(2) 

4 

-  z(2)     X 
zl     ,|A2 

(2)        >J 
2 

(2) 
x2 

xi  =  : 

,      X, 
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IV.   THE  MAXIMUM  PRINCIPLE  AMD  THE  VARIATIONAL  TECHNIQUES 

In  preceding  chapters  the  variational  principle  has  been  used  to  derive 
the  various  forms  of  the  maximum  principle.  There  are  several  other 
optimization  methods  and  techniques  which  are  based  on  the  variational 
principle  and  related  techniques.  Two  of  the  better  known  methods  are  the 
calculus  of  variations  and  dynamic  programming. 

In  this  chapter  the  interrelationships  among  the  various  methods  together 
with  some  general  aspects  of  the  variational  principle  are  discussed. 

The  Maximum  Principle  and  the  Calculus  of  Variations  (10) 

In  this  section  the  well-known  fundamental  necessary  conditions  in  the 
calculus  of  variations  are  derived  from  the  maximum  principle  when  the 
decision  vector  is  not  constrained.  Conversely,  by  using  the  calculus  of 
variations  techniques,  the  weakened  form  of  the  maximum  principle  can  be 
derived.  It  is  worth  noting  that  the  calculus  of  variations  is  often  frus- 
trated in  solving  problems  when 

a.  there  is  linearity  in  the  decision  variables, 

b.  they  are  two-point  boundary  value  problems, 

c.  there  are  unusual  functions, 

d.  there  are  inequality  constraints  on  the  decision  variables  (11).* 

1.  The  fundamental  problem  of  the  calculus  of  variations.  The  problem 
may  be  formulated  as  follows  (15) 

*  There  have  been  some  successful  attempts  to  extend  the  classical  calculus 
of  variations  to  the  case  where  the  decision  variables  are  constrained  (12, 

(13.  1/0. 
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A  real  function 

F(t,  x,(t),  x2(t) xs(t);  e1(t).  e2(t) eg(t))  =  F(t,  x(t),  e(t)) 

is  defined  in  some  region  R  of  the  space  of  the  real  variables  t,  x-^,  x^,..., 
x  for  arbitrary  real  finite  values  6,,  92,...,  ©s.  The  function  F  is 
continuous  in  all  its  arguments.  One  shall  consider  the  collection  of  all 
piece-wise  smooth  curves 

%-Xjft),     i  =  l,  2,...,  s,     tQ<t<T,  (1) 

lying  in  the  region  R  and  joining  the  points 

(t0,  x(0))  =  (tQ,  av  a2,...,  as)   =  (t  ,  a)  ,  (2) 

and  ' 

(T,  x(T))  =  (T,  Plt  P2 3s)  -  (T,  p)  .  (3) 

Along  each  such  admissible  comparison  curve,  the  objective  function  in  the 
form 

S-  T  F  (t,  x(t),  x(t))  dt  (4) 


•I 


0 
has  a  trell  defined  value. 

The  problem  is  to  find  the  curve  (or  the  extremal)  such  that  the 
objective  function  has  an  extremum. 

The  functions 

x  (t),    i  s=  1,  2,...,  s 

are  assumed  to  be  absolutely  continuous  and  to  have  bounded  derivatives, 

that  is,  at  every  point  Where  the  derivative  exists, 

dx,(t) 

— — —  <  M  (constant) ,    1*1,2 s;    t  <  t  <  T.   (5) 

The  set  of  all  absolutely  continuous  curves 


x=  (x1(t),  x^t),...,  xs(t)),     tQ  <  t  <  T 


^7 


are  in  a  6-neighborhood  of  x^t)  if 

xAt)  -  xAt)    <  6,     for  tQ  <  t  <  T,     i  =  1,  2,...,  s 

2.   The  Euler-Lagrange  equation  and  Legendre's  necessary1-  condition* 
Consider  the  following  specific  set  of  differential  equations 

dx,(t)  ,« 

^(t)  =  — i_-  =  ei(t)  ,    1-1,2 s  (6) 

and  the  objective  function 
T 

s  =  |  F(t,  ^(t),  XgCt),...,  xs(t);  e1(t),  e2(t) eg(t))  dt 

T 

=  f  F(t,  x(t),  e(t))  dt 

which  is  to  be  minimized  or  maximized. 

Using  Equation  (6),  one  has 

T 
S=  f  F(t,  x(t),  x(t))  dt  .  (7) 


" 


0 
The  decision  vector  9(t),  t  <  t  <  T,  which  is  assumed  to  be  piece-wise 

continuous,  and  the  corresponding  absolutely  continuous  trajectory  x(t)  of 

the  system  represented  by  Equation  (6)  together  with  the  boundary  conditions 

given  by  Equations  (2)  and  (3),  xdll  be  called  optimal  if  there  exists  a 

5  >  0  such  that 

S  [x(t),  6(t)J>  S[x(t),  e(t)j  , 

or 

s  j_x(t),  e(t)j<  s  [x(t),  e(t)j 

for  every  decision  6(t)  for  vfcich  the  corresponding  trajectory  x(t)  lies  in 
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the  6-neighborhood  of  the  curve  x(t).  In  the  former  case  the  objective 
function  attains  its  minimum  and  in  the  latter,  the  maximum. 

Since  the  maximum  principle  is  a  necessary  condition  for  optimality, 
it  is  at  the  same  time  a  necessary  condition  for  the  curve  x(t)  to  be  an 
extremal  of  the  objective  function  represented  by  Equation  (7).  The  funda- 
mental problem  of  the  calculus  of  variations  is  the  one  in  which  final  time 
is  specified  and  both  end  points  are  fixed. 

In  order  to  obtain  the  Euler-Lagrange  equation  from  the  maximum  principle, 
an  additional  state  variable  xo+1  is  introduced  such  that 

xs+1(t)  =  -ggli  =  F(t,  x(t),  e(t));   xs+1(0)  =  0  .        (8) 

Thus  the  Hamiltonian  and  the  adjoint  system  take  the  form 

s+1 
H  =  S  z.x. 
i=l  l^ 

s=ziei  +  z2e2  +  '"  +zs0s  +  VlF  '  (9) 

dZi  _       SH     _       .         3F(t,  x,   6)  i  -  i     2  a  (10  ) 

at  dx.  s+1  ox. 

dz   .- 

-£&  =  o  .  (li) 

dt 

Assuming  that  the  optimum  lies  in  the  interior  portion  of  the  admissible 
region  of  G(t),  the  optimal  condition  is  determined  from  the  following: 

•zz-  =  0  =  z.   +  2„jj,  — So  —  »    i  =  1,  2,...,  s       U^; 


39.^   w  "  *i  T  "s+1 


Since 


zs+1(T)  =  1  . 


From  Equation  (11)  one  thus  has 

zs+1(t)  =  1  .  (13) 
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Hence  Equation  (12)  is  reduced  to 


/+  \  _   oF(t,  x,  6)  _   oF(t,  x,  x) 
tt;  -  "    39,     -  "    ox, 


(14) 


Substituting  Equation  (13)  into  Equation  (10)  and  integrating  the  resulting 
equation  give 


2.(t)  =  2.(t0)  -  r  5F^x>  e>  dt,     i  =  i,  2 s. 

Combining  Equations  (14)  and  (15) .  one  obtains 


(15) 


g&J£»  9>  =  f  £&-*L*l  dt  -  2.  (t0), 
ox-;    i    ax-,      l  w 


t     **1 


2.  — *  J_  >  *-  j  •  •  •  }  S 


(16) 


•which  are  the  Euler-Lagrange  equations  in  the  integral  form.  Differentiating 
Equation  (16)  with  respect  to  t  yields 

(17) 


9F(t,  x,  9)   _d_  ,oF(t,  x,  9)x  _  n 


1  —  -L  >  £  |  •  *  •  >  S 


which  are  the  Euler-Lagrange  equations  in  the  usual  form. 

In  solving  problems  by  using  the  calculus  of  variations,  the  existence 
and  continuity  of  all  partial  derivatives  of  F(t,  x,  x)  up  to  the  fourth 
order  are  assumed  (16).  Then,  if  the  Hamiltonian  attains  its  minimum  as  a 
function  of  9(t),  the  quadratic  form 


s 

E   2 
JW.  j=l 


%    Sfl&l  &Ml  x(t)t  9(t)) 


39i  39j 


e  =  e 


hh-"' 


It  follows  from  Equations  (9)  and  (13)  that 


|   |  32?(t,  x(t),  6(f)) 

arf.  j=i   sei  S9j 


E.  g.  >  0  for  all  t,  t  <t<T  (18) 
-i  5j  -  0 


9  =  9 

where  §  is  an  arbitrary  constant.  This  condition  which  is  necessary  for  the 


curve  x(t)  to  be  an  extremal  for  the  minimum  objective  function,  is  called 
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Legendre's  necessary  condition. 

3.   T'feierstrass  necessary  condition.  According  to  the  maximum  principle 
the  necessary  condition  for  a  minimum  objective  function  is  that 

H(t,  x(t),  z(t),  e(t))  >H(t,  x(t),  z(t),  e(t))  .       (19) 

Using  Equation  (9)»  we  consider 

H(t,  x(t)t  z(t)f  e(t))  -  H(t,  x(t).  z(t),  e(t)) 

s 


2     (Q,    _  Q.)  m(t'  x(t)_'   zCt)'  e(t)) 


i=l      x        1  00, 


=  zs+1  |_F(t,  x,  e)  -  F(t,  x,  e)j  +  z   zi(ei  -  e±) 


i=l 
s 


-  A  \  -  V  ft^  Vi + «J 


i=l      *        *    *        B6i 


■  zs+i  lFCt.x.e)  -  F(t,x,9)J  -  s    (et  -  e.)  *H*»*r  e)j,     (20) 

i=l  B0. 

Substituting  Equation  (13)  into  Equation  (20)  gives 

H(t,  x,  z,  e)  -  H(t,  x,  z,  0)  -  s  (e.  -  0.)  ^H(tf  x,  z,  e) 

i=l   1    x      o8i 

=  {I  F(t,  x,  e)  -  F(t,  x,  0)  j  -  2  (e  -  9.)  M(t,jc,  GH  .   (21) 

^  J   i=l   x    x     o9±   J 

In  the  calculus  of  variations  the  Weierstrass  E-function  is  defined  as 


E  =  F(t,  x,  6)  -  F(t,  x,  9)  -  S  (8.  -  9.)  *?&'  x'   6)  .       (22) 

irf.   1    1     o9\ 

If  the  Hamiltonian  function  attains  its  maximum  at  an  interior  point  of  the 

region  in  which  the  decision  vector  9(t)  (or  x(t))  is  defined,  one  has 

5H(t,  x,  z?  9)  m  Q  t     .=1>2 s< 

09, 
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Combination  of  Equations  (19),  (21),  (22),  and  (23)  yields 

E  >  0  .  (24) 

which  is  the  Weierstrass  necessary  condition  for  a  minimum  objective  function. 
It  is  worth  noting  that  Equation  (24)  is  not  applicable  if  the  decision 
vector  G(t)  lies  on  the  boundary  of  the  region  defined.  However,  the 
maximum  principle  does  not  have  this  deficiency  (10). 

4.   The  problem  of  Bolza.  The  problem  of  Bolza  as  formulated  by 
Bliss  (16)  is  stated  as  follows: 

It  is  desired  to  find  in  a  class  of  curves 

^(t)  ,     1-1.  2 s,     tQ  <t  <  T 

satisfying  the  differential  equations 

§.(t,  x(t),  x(t))  =  0,     j  =  1,  2,...,  m<  s  (25) 

and  the  end  conditions 

*k(V  x(V*  T»  x(T))  =  °'     k  =  1* 2 p-2s  +  2     (26) 

a  curve  which  minimizes  an  objective  function  of  the  form 

S  =  g  [t0.  x(tQ),  T,  x(T)j  +  f  F(t,  x(t),  x(t))  dt  .       (27) 

It  will  be  shown  that  the  optimization  problem  which  has  been  treated  so  far 
is  equivalent  to  the  problem  of  Bolza  in  x-Mch  either  g  =  0  or  F  =  0,  that 
Is,  it  is  equivalent  to  the  problem  of  Lagrange  or  the  problem  of  Mayer. 
Consider  the  following  system  of  differential  equations 

dX.  ^rn+l       ^c 

^=f.(t.  x1(t),  XjCt) xs(t).  -^i -gS)  ,      (28) 

i  =  1,  2,...,  m<s 

and 

-X|ii=e;.,        j  =  1,  2,...,  s-m.  (29) 
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*i   .  ^  -  ~  -  -    ., ,  (t).  Ss* J, 

s  dt  dt 

dx   .,  dx 


111+1 t^)  =  0  ,  (30) 


Equation  (28)  can  be  rewritten  as 

=  ^(t.  ^(t) is(t).  dt  dt 

i  =  1,  2, . . . ,  m  <  s. 

This  is  equivalent  to  Equation  (25). 

The  boundary  conditions  of  the  optimization  problem  are  usually  of 
the  form 

x(t0)  -  a  ,  x(T)  =  0  (3D 

which  is  equivalent  to  Equation  (26).  The  problem  is  to  find  an  admissible 
optimal  control  (or  the  optimal  decision)  6(t)  such  that  the  corresponding 
trajectory  of  the  system  satisfies  Equations (30 )  and  (31)  and  that  the 
objective  function 

S  =  J  F(t,  x(t),  e(t))  dt, 

x&ich  can  be  obtained  from  Equation  (2?)  by  setting 

g  =  0  , 
attains  its  minimum.  It  may  be  noted  that  the  problem  in  -which  the  objective 
function  is  of  the  form 

s 

S  =  S  ex.  (T) 
i=l  X  X 

has  also  been  treated  before.  This  is  equibalent  to  the  case 

F  =  0 

in  Equation  (2?).  The  values  of  tQ  and  x(tQ)  are  usually  given.  x(T)  is  to 

be  determined  and  T  may  be  fixed  or  unspecified.  It  may  also  be  noted  that 
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problems  in  which  the  objective  function  is  in  the  integral  form  can  be 
reduced  to  the  form  of  the  linear  combination  of  the  final  state  variable 
by  introducing  an  additional  state  variable. 

5.  From  the  calculus  of  variations  to  the  maximum  principle.  The 
maximum  principle  can  be  derived  by  using  the  classical  calculus  of  variations, 
if  the  decision  variables  are  not  constrained. 
Suppose  that  a  piece-wise  smooth  curve 

X-x(t),     tQ  <  t  <  T 
lies  entirely  in  the  space  of  the  real  variables  t,  x,,  x2,...,  x  and  gives 
the  objective  function  S  a  weak  relative  extremum.  A  piece-wise  smooth 
vector  function 

y(t)  =  (y1(t),  y2(t),...,  ys(t)) 
is  chosen  such  that  it  satisfies  the  boundary  conditions 

y±(t0)  =  0,    i  =  1,  2 s.  (32) 

Consider  the  equation 

x(t)  =  x(t)  +  ey(t)  .  (33) 

If  e  is  sufficiently  small,  the  function  x(t)  lies  in  a  neighborhood  of 
the  extremal  x(t).  The  corresponding  equation  for  Q(t)  is 

9(t)  =  e(t)  +  ecp(t).  (3*0 

Then  the  objective  function  can  be  considered  as  a  function  of  e,  that  is 

S(x(t)  +  ey(t))  =  ♦(«).  (35) 

and  it  attains  its  extremum  when  e  =  0.  Consequently,  the  condition 

*'(0)  =  f     =0  (36) 

6=0 

provides  the  necessary  condition  for  the  extremum.  The  objective  function 
which  is  to  be  minimized  is  of  the  form 
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s 
S  =    2     c.  x.  (T). 

i=l    x    x 

(37) 

It  can  be  written  in  the  form 

T    s                            s 
S  =  f      S    c,  x,  dt  +    2     c,  x.(t  ). 
l     i=d    X    x            i=l     i    X     ° 

(33) 

Tne  performance  equations  are 

dx. 

5^  =  ^(t)  =  fjft,  x(t),  e(t)),        l-l,  2 s 

(39) 

or 

^(t)  -  f±(t,  x(t),  6(t))  =  0,           i  =  1,  2 s. 

(40) 

Employing  -  zi(t)  as  the  Lagrange  multipliers,  one  has 

s-  r    1   s    *X  -    2    z,(t)(x.(t)  -f.(t,  x(t),  9(t))jdt 
t0  Li=l    X  1      i=l    1          x              X                            J 

+  Jlci\V- 

(41) 

Differentiating  the  above  equation  with  respect  to  e  gives 

T      s                            dy.  (t)        s      s              Sf .  (t,  x, 

f«(0)-0-J    |    2     (c,   -  z,(t))  — i---    2      2    a  (t)~i — 

l0  Li=l      1        x              ^          i=1  J=1    i               Bx^ 

,  e) 

y.Jdt 

T      s      r          of.  (t,  x,  6) 

"       1      *****       -               1                                  rn           H+    —    0 

(42) 

"  i0  Lw  >a  "*        ^      **J     "    • 

where  r  is  the  number  of  components  of  the  decision  vector. 

Integrating  by  parts  the  first  term  in  the  first  bracketed  quantity 

on  the  right  hand  side  of  Equation  (42),  one  obtains 

s                                      T        T      s  dz.(t)     s      s              of  .(t,  x, 

-  1    2  (c.  -  z.(t))y  (t)J      -  f        2  -i-  +  22  >  (t)  — ^ 

Lfc=l    *        *          x      J  t      t    Li=l  dt        i=l  j=l  J                ox. 

oo                                           1 

8) 

>.dt 
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T   s     r  af. (t,  xf  e) 

+  f      2      2     z,(t)— i <p    dt.  (43) 

t0i=ij=i  a§j        J 


Applying  the  boundary  conditions,  Equation  (32),  to  Equation  (43)  gives 

s  T      s    dz,(t)        s      s  3f.(t,x(t),6(t)) 

-  I    2  (c.  -  z,(T))y.(T)    =  J        2    -|r-  +22  z  (t)-J Jy  (t)dt 

L1p0.    ^        x  x      J      trt  Li=l      dtr  i=l  j=l  a  Sx.  "ri 


+  f      2      2    z.  (t)  —i— i — 1— — —  cp    dt.       (44) 
trt  *d    <-">     x 


T    s      r                Sf.(t,  x(t),   e(t)) 
2      2    z.(t)  —* 

Since  7.«(t)  and  cpi(t)  are  arbitrary  on  the  interval  tQ  <  t  <  T,  in  order 
for  Equation  (44)  to  be  an  equality,  the  following  conditions  must  be 
satisfied 

ci  -  Zj_(T)  =  0  , 


i  =  1,  2 s    (45) 

3f,(t,  x(t),  §(t)) 

't)-J :: =o. 

dt    j=l  J         B^ 


dz,(t)   s     3f.(t,  x(t),  9(t)) 

+  2  z,(t)  —2 =  0, 


and 


or 


s     sf,(t,  x(t),  e(t)) 

2  z.(t)-J =0,     0  =  1,2 r      (46) 

i=l  x         o6. 


Zi(T)  =  c±  ,     i  =  1,  2 s  (47) 

dz,     s      of,(t,  x(t),  6(t)) 

~=-  2  z,(t)  -J ,   i  =  l.  2,...,  s    (48) 

dt     0=1  °         5% 


and 


s    of.(t,  x(t),  e(t)) 

2  z  _1 p    j  =  1.  2,...,  r.  (49) 

i=i  x     se. 

Equations  (47)  and  (48)  are  the  adjoint  system  in  the  maximum  principle  and 
Equation  (49)  is  the  necessary  condition  for  the  extremum.  It  is  noted  that 
the  derivation  used  in  this  section  leads  only  to  the  stationary  condition 
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of  the  maximum  principle  (18). 

6.   The  canonical  equations  and  transformations.  There  are  many- 
different  -ways  by  which  coordinates  can  be  transformed.  In  general  the 
objective  of  a  transformation  is  to  change  an  original  system  of  equations 
to  a  new  system  of  equivalent  equations  such  that  the  latter  system  is 
easier  to  handle  and  clearer  to  visualize  than  the  original  form.  The 
transformation  uhich  has  been  used  is  equivalent  to  the  one  frequently 
applied  in  the  classical  mechanics.  In  this  section  a  transformation  -which 
leads  to  the  algorithm  of  the  maximum  principle  is  presented. 

Suppose  that  an  objective  function  of  the  form 

T 
S  =  J  F(t,  x(t),  S(t))  dt 

is  to  be  minimized,  -where 

e±(t)  =  ^=  *t(t)  »    1-1.2 s. 

Define  the  Hamiltonian  function  and  the  adjoint  variables  as  follows: (19): 

H-  2  ».   x.  +8  ..  F(t,  x(t),  e(t)) 
fc=l  x  *■        s+1 

=  2  zixi+F(t,  x(t),  e(t)).  (50) 


fed. 


and 


3F 


a.  (t)«-—  ,     i  =  l,  2,...,  s.  (51) 


1       ax. 


Assuming  that  the  Jacobian 

a  (z, ,  zof...,  z  ) 

1—1 -s-^  o  . 

O  (X,  »  x_  ■ » •  • »  x  ) 
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we  can  make  a  local  transformation  from  the  variables  t,  x-,,  X£ x  , 

x.,  x~f»,   x  ,  F  to  the  variables  t,  x,,  x„ x  ,  z,,  z2».^.»  »  i  H, 

which  are  called  canonical  variables. 

Hence  the  Kamiltonian  and  adjoint  system  of  the  maximum  principle  can 
be  derived  by  using  the  definitions  of  Equations  (50)  and  (51)  •  From 
Equation  (50)  one  has 

dH  =  2  z,  dx.  +  2  x,  dz,  +  |?  dt  +  S  f£-  dx ,  +  E  ft-  dx..    (52) 
^  i  x  i=1  i  i  St    i=i^  i  i=1 a^  i 

Substituting  Equation  (51)  into  Equation  (52)  gives 

s  s 

dH  =  2  x.  dz,  +  ~  dt  +  E  f§-  dx,.  (53) 

i_1  i   i   St     1_1  S^   i 

It  follows  from  Equation  (53)  that 


SH_  3F 

st  at  • 

5H    5F 

ax.  ax.  * 

_u  ==  J-  >  i-  r  •  •  *  j  S 


(54) 
(55) 


an   •  _  ^  /^n 

az~  =  xi  ~  dT  •  (56) 

The  Euler-Lagrange  equations  for  the  objective  function  S  are 

j=r-4f£--o  ,    i  =  l,  2 s.  (57) 

a^  dt  ^ 

Combining  Equations  (51)  .  (.55),   (5^)»  and  (57)   gives  the  following  set  of 
equations  called  the  canonical  Euler  equations  (19) 

If  the  Hamiltonian  function  does  not  depend  on  t  explicitly,  we  have 
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Substituting  Equations  (58)  and  (59)  into  Equation  (60)  yields 

A.  I    *.A-.A.'fL).o.  (6i) 

dt   3_1  Naxj_  3z±   oxj^  3zi 

Therefore,  it  follows  that  H  is  constant  along  the  extremal. 

The  transformation  from  the  variables  t,  x,  x,  and  the  function 
?(t,  x,  x)  to  those  of  t,  x,  z,  and  H(t,  x,  z)  respectively  is  called  the 
Legendre  transformation  (19).  If  H(t,  x,  z)  is  subjected  to  the  Legendre 
transformation,  the  function  F  is  recovered  as  follows: 

Using  Equations  (50)  and  (56),  we  have 

s         s  s 

H  -  2  z,  H-=  2  z,  x.  +F(t,  x(t),  x(t))  -  £  z,  x, 
fed  x  3zi   i=l  x  x  WL  i  x 

=  F(t,  x(t),  x(t)). 

This  indicates  that  the  Legendre  transformation  is  its  own  inverse. 

7.  The  transversality  conditions.  Suppose  that  the  objective  function 

is  of  the  form 

T 
S  =  J  F(t,  x(t),  x(t))  dt  ,  (62) 

and  that  the  initial  point  is  fixed  at 

x(tQ)  »  or  , 
and  that  the  final  point  lies  on  the  given  hypersurface 

h(x(T))  =  0.  (63) 

Let  T  be  the  final  time  when  the  optimal  trajectory  hits  the  given  surface. 
One  then  has 

T  =  T  +  6T  , 
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and  the  variational  equation  is 

x(t)  =  x(t)  +  ey(t). 
The  difference  of  the  objective  function  is 
JT+6T 


AS  =  f    F(t,  x+ey,   x+ey)  dt  -  /  P(t,  x,  x)  dt 


=  f  [?(t,  x+ey,  x+ey)  -  F(t,  x,  x)J  dt 


(64) 


T+6T 
+  J  F(t,  x+ey,  x+ey)  dt  . 

T 

Hence  the  variation  of  the  objective  function  becomes 

J  s 


roF 


oF 


6S  =  {0^(%e7i  +  %^)dt+F 


6T. 


t=T 


(65) 


(66Y 


Integrating  the  second  member  of  the  first  term  in  Equation  (66)   by  parts 
gives 


6S  = 


88  f  =  (lr  -  ft  §■)  e^ict> dt  +  =  S1  v±(t) 

t  i=l    dxi      dx>  Sx  *  id.8^      x 


+  F 


6T.        (6?) 

t=f 


It  can  be  seen  from  Figure  8  that 


ey,  (t) 


_  =  6x^(T)  -  x±(T)  6T. 


Substituting  Equation  (68)  into  Equation  (67)   yields 

6S  =  /T  2  (g-   d  ^L)ey  (t)  dt  +  (F  -  2  x.^)6t 
lQ  fed  3xi   dt^   i  i=l  x  0% 


t=T 


(68) 


jr  [_ey(t)J  =  ey(t),  that  is,  the  derivative  of  a  variation  is  the 


variation  of  a  derivative  (17). 


i< 


tc 


— iC 


—  B 


D(T+6T,X(T)+&x(T)) 


T+T+ST 


-^t 


AE=eyi(T)|tcT       BD  =  ^Xi(T) 
CD=x-.(T)5T  AE=BD-CD 


Fig.8.  Variation  of   the   trajectory. 
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+  s  2L-, 


(69) 


Since  the  objective  function  S  has  an  extremum  along  the  extremal  x(t),  the 
Euler-Lagrange  equation  must  be  satisfied,  that  is 

X  —  X  j  ^- y  •  •  •  t    s» 


oF    d  oF    Q 
^i  '  dt  aii  ~   ' 


It  follows  that 


6S  =  (F  -    i    x    ^-)  fit 

k=l        9^ 


~-  6x 


t=T      i=l  Sii 


t=T 


Substituting  Equations  (50)  and  (51)  into  Equation  (70),  one  obtains 


6S  =  (  Z  z.  6x.  -  H  6t) 


fc=l 


i   a. 


t=T 


Thus  the  necessary  condition  for  an  extremum 

6S  =  0 
takes  the  form 


(70) 


(71) 


S  z.  6x.  -  H  6t  =  0, 

i=l  x   1 


(72) 


at  t  =  T.  According  to  the  theorem  of  Pontryagin  for  final  time  unspecified 
(see  Appendix  1),  if  S  attains  a  minimum 


H  .  =0. 

man 


Equation  (72)  then  becomes 


s 

S  z.  6x.  =  0  , 
i=l  1   x 


(73) 


at  t  «  T. 

If  the  vector  a  =  (a^,  a2»»».»  a  )  belongs  to  the  tangent  plane  of 
h(x)  -  0  at  t  =  f,  Equation  (73)  can  be  written  as 
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2  z.  a.  =  0.  (7*0 

i=d  x  i 

Similarly,  if  the  initial  point  x(tQ)  lies  on  the  hypersurface 

h(x(tQ))  =  0, 

one  can  obtain  the  similar  transversality  condition 

s 

2  z.  a.  =  0, 
i==l  1  1 

at  t  =  tg. 

8.  The  Hamiltonian-Jacobi  equation  (19 )»  If  in  Equation  (71) »  one 

defines  a  new  function 

s 
dW  =  Z  z,  dx.  -  H  dt  (75) 

i==l  i   x 

at  t  b  T,  then  it  follows  that 

aw 


S^""! 


z.,         1  ■  1,  2,,..,  i  (76) 


f=-H.  (77) 

From  Equation  (77) »  one  has 

gf  +  H(t,  x(t),  §^)-0  (78) 

at  t  ■  T.  Equation  (78)  is  known  as  the  Hamiltonian-Jacobi  equation.  Let 
V*  W(t,  x(t),  n)  (79) 

be  a  solution  of  the  Hamiltonian-Jacobi  equation,  where 
fl  =  (!)■,»  Tl  , . .  • ,  11  ),  m  <  s  is  a  parameter. 


Then 


ii  (of*}  =  °*      i  =  1» 2 m-  (80> 
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This  can  be  shown  as  follows:  . 
Since  one  has 

±  (siL)  =  _A_+2     is.  Sa.  (si) 

dt  va%;  st  S%  «  3xj  a%  dt  v  ' 

Substituting  Equation  (79)  into  Equation  (78)  and  differentiating  it  with 
respect  to  Tl^  give 

-  J&t  ,»'.{    fL  JjfiL  .  o.  (82) 


From  Equations  (81)  and  (82)  one  thus  obtains 

dt  ^Tl±;  =  ^  3Z;j  BXj  o\  "  ^  oxj  3%  dt 

=  |    5  g   !  <2L  -  ~i  j  .  o  .  (83) 

along  each  extremal.  Comparing  Equations  (61)  with  Equation  (83),  one  sets 
that  both  H  and  |~-  are  the  first  integrals  of  the  canonical  Euler  equations. 

The  Maximum  Principle  and  LJ/namic  Programming 

It  will  be  shown  in  this  section  that  there  exists  a  close  relation 
between  the  maximum  principle  and  the  method  of  dynamic  programming. 

The  method  of  dynamic  programming  is  based  on  the  principle  of  optimality 
stated  by  Bellman  (1)  as 

"An  optimal  policy  has  the  property  that  whatever  the  initial 
state  and  initial  decision  are,  the  remaining  decisions  must 
constitute  an  optimal  policy  with  regard  to  the  state  resulting 
from  the  first  decision." 
It  is  desired  to  minimize  an  objective  function  of  the  form 


<& 


s 
S=  E  c,  x,  (T  ), 

i*l  X      1 

with  the  initial  condition 

x(t0)  =  or, 

and  with  the  final  time  T  fixed. 

The  principle  of  optimality  implicitly  states  that  the  minimum  value 

of  S  is  a  function  of  the  initial  state  x(tQ)  =  a,  and  the  initial  time  tQ. 

Introducing  the  function  (20) 

s 
S(x(t0),  t0)  -  Mia  S  -  lfln  Z  c^  x^Cr),    t0  <  t      (84-) 

e(t)   e(t)  jwl 

and  denoting  the  optimal  trajectory  at  t  =  tQ  +  At  by  x(t),  where  At  is 

sufficiently  small,  one  has 

S(x(tQ),  tQ)  =  S(x,  t).  (85) 

Assume  that  the  trajectory  from  tQ-tAt  to  T  is  optimal.  Then  for  any  other 

trajectory  x(t)  at  tQ+At,  we  have  S(x,  t).  From  the  definition  of  Equation 

(8*0,  it  follows  that 

S(x(tn),  tn)  =  tan  S(x,  t),    t0  <  t  <  tn-tAt.  (86) 

0   0   Q(t)  -  -  0 

Assuming  the  existence  and  continuity  of  the  partial  derivatives  of  s(x,  t), 
it  can  be  expanded  in  a  Taylor  series  to  give 

Sfe.  t)  =  S(x(t0),  tQ)  «  fL   (Xj_  .  Xi(to))  +  ||-  (t-t0)  +  0(s2),   (87) 

where  the  partial  derivatives  are  evaluated  at  (x(tQ),  tQ),  and  0(e) 

2  ?     9 

represents  the  terms  of  e  and  those  of  higher  orders  than  e  .  0(0  should 

satisfy  the  condition  that 

Dm  slM-o. 
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The  performance  equations  are  again  given  by 

dx. 
x±(t)  -55*-  ft(t.  x(t),  e(t)),    i  =  1,  2 s. 


Hence 


^(t)  =  ^(t0)  +  f±(t0,  x(tQ),  9(t0))  At  +  0(e2), 


(88) 


(89) 


Substituting  Equations  (87)  and  (89)  into  Equation  (86)  gives 
S(x(t  ).  t  )  =  **    I  S(x(t  ),  t)+s|  f  At 

0   °   e(t)  L   °   °   i^i  **±   x 

+  ||-  At  +  0(e2)j  ,   tQ  <  t  <  tQ+  At.  (90) 

The  decision  vector  consists  of  those  terms  containing  fj_.  By  simplifying 
Equation  (90)  and  dividing  it  by  At,  one  obtains 
cS(x(tJ,  tn)        ,  s  xq      n,  2, 


«-^UIK+W  vs^v? 


At 


ato      e(t)  Li^i  Sxi 

Letting  At  approaches  zero  yields 

5S(xto)'  V  -  -  «■  Effi  (t0.  x(t0),  e(t0)).        (91) 
*h  e(t)  i^  Bxi  x  °   ° 

at  t  =  tn.  This  determines  the  choice  of  0(t)  at  t  =  tQ.  Equation  .(91)  is 
valid  for  any  tQ;  it  can  be  applied  to  any  point  in  the  interval  |_tQ,  TJ. 
Therefore,  it  can  be  written  in  general  as 


oS(x(t),  t)  m  _    ^   2  |i-f.(t,  x(t),  G(t)).  (92) 

at       8(t)  i=l  ^  x 

It  will  be  shown  that  this  is  equivalent  to  the  optimal  condition  given  by 

the  maximum  principle.  Let 

Zi(t)  =  _  MkMi_ti  f   z(T)  =  Ci,    i  =  l,2 s    (93) 

and 
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3S(x(t),  t)  m  H(t>  -(t)>  5(t)|  z(t)) 

=  s  z.(t)  f.(t,  x(t),  e(t)).  (9*0 

i=i  l    x 

From  Equations  (93)  and  (9*0  one  has 


dz.     s    df 


i 


=  -  E  z. 


±1  (95) 


dV    j=l  J  5xi 
then  combining  Equations  (92),  (93).  and  (9*0  one  obtains 

H(t,  x(t),  e(t),  z(t))  =  Mia  H(t,  x(t),  e(t),  z(t)).         (96) 

e(t) 

The  relationship  between  the  maximum  principle  and  dynamic  programming  is 
analogous  to  that  between  the  Hamiltonian  system  and  the  Hamiltonian-Jacobi 
equation  [compare  Equations  (93).  (9*0.  and  {96)   with  Equations  (76),  (77), 
and  (78)j. 

Generally,  in  order  to  find  the  optimal  trajectory,  the  method  of 
dynamic  programming  requires  the  exhaustive  stepwise  calculation  by  using 
Equations  (88)  and  (91),  and  the  objective  function 

s 

S  =  2  c.  x.  (T). 

i^l  *  x 

However,  using  the  technique  of  the  maximum  principle,  one  needs  the  solution 
of  the  differential  equations,  Equations  (88)  and  (96),  and  the  optimal 
condition,  Equation  (96),     The  advantages  and  the  shortcomings  of  each 
method  have  been  broadly  discussed  elsewhere  (10,  11,  6,  21). 

It  is  worthwhile  to  mention  that  the  method  of  dynamic  programming 
results  in  a  set  of  partial  differential  equations,  whereas  the  maximum 
principle  gives  a  set  of  ordinary  differential  equations.  Th.e  method  of 
characteristics,  however,  can  be  used  to  transform  the  set  of  partial 
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differential  equations  to  a  set  of  ordinary  differential  equations  (11). 
Dynamic  Programming  and  the  Calculus  of  Variations 

In  the  calculus  of  variations,  we  consider  the  properties  of  the 
admissible  curves  lying  in  the  6 -neighborhood  of  the  optimal  curve  and 
hence  obtain  all  the  necessary  conditions.  By  using  the  method  of  dynamic 
programming,  instead  of  finding  the  whole  extremal,  one  evaluates  the 
optimal  derivative  point  by  point  along  the  extremal.  It  will  be  shown 
how  the  working  equations  of  the  calculus  of  variations  can  be  derived  by 
using  dynamic  programming. 

Consider 

Sp-ijW-ejCt),   i.=  i,  2,...,  s.  (97) 

It  is  desired  to  maximize  the  objective  function  S  of  the  form 

T 
S  =  J  F(t,  x(t),  x(t))  dt. 

The  boundary  conditions  are 
x(tQ)  =  a  , 
x(T)  =  0. 
Define  the  function 


S(t,  x(t))  =  Min  r  F(t,  x(t),  9(t))  dt  ,  (98) 


T 

e(t) 

where  S(t,  x(t))  is  the  minimum  value  of  the  integral  of  F(t,  x(t),  9(t)) 
from  the  point  (t,  x(t))  to  the  fixed  point  (T,  x(T)).  It  is  clear  that 
S(T,  x(T))  =  0. 
Breaking  up  the  time  interval  It,  TJ  into  two  parts,  It,  t+Atj  and 
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jt+At,  t],  Equation  (98)  can  be  written  as 


t+At 
S(t,  x(t))  =      Mia  ran  if'  F(t',  x(t'),  e(t'))  dt' 

e(t)         e(t)      4; 

t<t'<t+At  t+At<t'<T 


+  J       F(t',~x(t>),  e(t«))jdt« 

t+At 


t+At 
Kin  F(t»,  x(t'),  e(t«))  dt' 

6(t)  t 

t<t '<t+At 

T 
+        ran         f        F(t«,  x(t'),  9(f))  dt  |  .  (99) 

e(t)     't+At  J 

t+At<t'<T 


According  to  the  definition  given  by  Equation  (98),  one  has 

T 
S(t+At,  x+xAt)  =        Mia  F(t',  x(t"),  e(t'))  dt'   . 

G(t)  t+At 

t+At<t'<T 

Substituting  this  equation  into  Equation  (99)  gives 

t+At  .  k 

S(t,  x(t))  -    ran  F(t',  x(t«),  9(f))  dt'  +  S(t+At,  x+xAt). 

e(t)   t 

t<t'<t+At  (100), 

For  sufficiently  small  At,  Equation  (100)  becomes 

S(t,  x(t))  =    Kin   I  F(t,  x(t),  6(t))At  +  S(t+At,  x+xAt)  I  +  I  0(At2)  I 
6(t)  L  J   l.     j 

t<t'<t+At  (101) 

which  is  the  functional  formulation  of  the  principle  of  optimality. 
A  Taylor  series  expansion  of  Equation  (101)  yields 

S(t,  x(t))  =   Min    |F(t,  x(t),  9(t))  At  +  S(t,  x(t)) 

e(t)   L 

t<t<t+At  . 
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||  At  +  2  P-X.   At]  +  0(At2).  (102) 

ot      .  •,  ox.   X   J 


Letting  At  approaches  zero,  the  following  nonlinear  partial  differential 
equation  is  obtained  (22): 

0  =  Kin  j_F(t,  x(t),  8(t))  +  ||  +  2  x±  g-j  (103) 

9  (t )  i?=l     i 

■which  is  an  equation  of  the  Hamiltonian-Jacpbi  type  and  is  called  the  Bellman 

equation.  Equation  (103)  is  equivalent  to  the  following  two  equations 

M-+||.«0,    i=l,  2,...,  s  (104) 

which  is  obtained  by  taking  the  derivative  of  Equation  (103)  "with  respect 
to  x.  and 

F  +  |f  +  S  x,-U-  =  0  ,  (105) 

at  ^  1&. 

which  is  the  minimum  value  of  Equation  (101).  Differentiating  Equation 
(104)  with  respect  to  t  gives 

Jt| +  5^1t  + 1,5^10  =  °-   *"»•» -    006) 

Partial  differentiation  of  Equation  (105)  with  respect  to  x.  (t)  yields 

X—   X|   *~  y  •  •  •  |   S 

which  may  be  reduced  to 

%  +  St%  +  j1ij3^=0'     1"1-2 s      •   O07) 

by  using  Equation  (104).  Substracting  Equation  (104)  from  Equation  (107) 
gives  the  set  of  Euler-Lagrange  equations 


. 
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M-.A^L-o     i,,  i      s 

(108) 

In  deriving  Equation  (10*0  the  fact  that  the  first  derivative  must  be 

zero  at  a  minimum  point  was  used.  Meanwhile,  at  the  minimum  point  the 

second  derivative  must  be 

*2p   >  0 

(109) 

dx.  dx.  "" 
1   j 

which  is  the  Legendre  necessary  condition  for  a  minimum. 

It  follows  from  Equation  (103)  that  for  the  optimal 

a* 

decision  x(t) 

(or  8(t))  one  has  the  following  inequality 

F(t,  x(t),  e(t))  +||+  1  i:ei^)  > 

F(t,  x(t),  G(t))  +||+  S  ff-QiCt)  . 

or 

F(t,  x(t),  e(t))  -  F(t,  x(t),  e(t))  +  s  (e  (t)  -  e 

1*L    1      2 

Ct»|L>.«. 

(110) 

Because  of  Equation  (104- ),  Equation  (110)  becomes 

F(t,  x(t),  e(t))  -  F(t,  x(t),  e(t))  -  s  (e  (t)  -  e 

i=i  x          J 

(t))^->o. 

(111) 

This  is  the  Weierstrass  necessary  condition  for  a  minimutr 

i.  Now  consider 

that  the  final  point  of  the  trajectory  lies  on  the  curve 

x=  g(T). 

(112) 

Eien  for  the  optimal  curve,  the  change  in  S  as  the  final  point  varies 

along 

the  given  curve  must  be  zero,  that  is 

BS  ,   y       oS      1     Q 

3t   iil  Sxi  dt 

(113) 
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at  t  as  T.  Combining  Equations  (104),  (105),  and  (113),  one  obtains 

F+  S  (  i.i)g-«0  ,  (114) 

fed  dt    x  Sx± 

which  is  equivalent  to  Equation  (70),  the  transversality  condition  at  the 

final  point. 

The  relationships  among  the  maximum  principle,  dynamic  programming, 

and  the  calculus  of  variations  have  been  discussed.  It  is  recognized  that 

there  is  no  single  optimization  technique  superior  to  other  techniques  in 

handling  every  type  of  problems.  How  to  choose  an  appropriate  technique 

to  solve  a  specific  type  of  problem  is  an  important  step.  The  selection 

of  a  proper  technique  depends  on  the  characteristics  of  the  problem,  the 

computing  facilities,  such  as  analog  or  digital  computers,  and  other  factors 

influencing  the  calculations  in  the  problem. 

The  Maximum  Principle  and  the  Adjoint  System 

There  are  several  ways  of  deriving  the  algorithms.  The  derivations 
using  adjoint  systems  and  Green's  functions  (23,  24)  will  be  discussed.  A 
numerical  iterative  technique  which  uses  the  Green's  function  will  also  be 
briefly  described. 

"Without  loss  of  generality,  one  can  consider  the  autonomous  system 
dx. 

25* «  \  - -fjXxft).  e(t))  ,   i  =  i,  2 s        (113) 

and  an  objective  function  of  the  form 

s 
S  =  Z    c.  x,  (T). 
irf.  x  ± 

Let  (x(t);  9(t))  be  the  optimal  point.     Then  the  variational  equations  are 
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x(t)  =  x(t)  +  Sx(t)  ,  (X&)* 

e(t)  =  e(t) +68(t).  (115)* 

Considering  the  variations  of  f^_  both  in  x^  and  8i  simultaneously  and 
linearizing  yields 

s  of.       r  of. 

6x.  =  2  ^r  6x,  +  2  >—  68,  ,     i  =  1,  2 s.      (116) 

1   j=l  Bxj   j   k=l  S8k   k 

The  partial  derivatives  are  evaluated  along  the  optimal  path.  For  the 

fixed  initial  point, 

6x±(t0)  =  0,        i  =  1,  2 s. 

The  so-called  adjoint  system  of  Equation  (116)  is  defined  by 

dz.  s  of . 

„2b«.L(t}«-  2  J*.  ,    i*l,  2 s.         (117) 

dt   i     ^  axi  o 

It  is  obtained  by  deleting  the  control  terms  and  transposing  the  matrix 
of  coefficients  and  changing  the  sign.  The  adjoint  vector  z(t)  may  also 
be  called  Green's  vector. 

Substituting  Equations  (116)  and  (117)  into  the  following  equation, 

S  S  3 

4r    2  z.  6x.  =  2  z.  6x.  +  2  z.  6x.  , 
dti=l  x      x   i=l  x      x   i=l  i   i 

gives 

.  s         s   r    of . 
£  2  a.  6x.  «  2   S  z.rrise..  (118) 

dt  fed  x  x   i=l  j=l  x  sej   J 

Equation  (118)  is  equivalent  to  Equation  (14)  of  Section  II.  Integrating 
Equation  (118)  from  t  =  tQ  to  t  ■  T,  one  obtains 

2     I  z.  (T)  fac  (T)  -  z.  (t  )  far  (t  )  Lf    I      2     z    -A  68     dt.        (119) 
i?lLl  x  l0         l    0J      tAi=l  j=l    x  3ej       J 


Equations  (114)  and  (115)  are  the  same  as  used  previously.  Here  6x(t) 
includes  the  terms  ey  and  0(e2)  and  68(t)  includes  eo  and  0(e2). 
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Suppose  that  the  boundary  conditions  of  z.  are  defined  as 


Since 


zi(T)  =  c.  ,        i  =  1,  2,...,  s. 


8*4 (*0)  =0.        i  =  1,  2,...,  s 


and  the  variation  of  the  objective  function  is  zero  along  the  optimal 
trajectory,  that  is 

6S=  2  c.  6x.(T)  =  0  ,  (120) 

Equation  (119)  therefore  becomes 

s  JT    s      r  of . 

i  .«     -  (121) 


E    c,   6x.(T)=  f      2      2    2    -i68     dt  =  0  , 

ri.      1        X  L    i=l   j=l      X  "j         J 


or 


s  5f .       >v 

This  is  the  necessary  condition  for  the  weak  form  of  the  maximum  principle 
(18)  (the  difference  between  the  weak  and  strong  forms  of  the  maximum 
principle  will  be  discussed  later). 

It  has  been  shown  that  it  is  also  convenient  to  use  Green's  functions 
(see  Appendix  2)  to  derive  the  above  conditions  (23,  2~,  25). 

A  solution  of  Equation  (116)  may  be  written  in  the  form 
s  tar  of. 

6Xl(t)  =  zo.  ,(Vt)  tej(t0)  + 1   £yt.T>  £  jef  6e*  dT         <122> 

where  G.  .(t,T)  are  called  Green's  functions  (or  influence  functions).  The 
first  term  on  the  right  hand  side  of  Equation  (122)  represents  solutions  of 
the  homogeneous  system  of  equations.  Green's  functions  transmit  the  influence 
of  a  unit  impulse  (or  Dirac  delta  function)  in  the  decision  variables  at 
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time  t,  or  unit  change  in  x. (0),  to  the  output  x. (t). 

Ifiiltiplying  both  sides  of  Equation  (122)  by  z^T)  and  summing 

s  s  T  s      r  of . 

Z  z. (T)  6x. (T)  =  Ez(t)6x(t)+J   2  z  (t)  Z  ^  56  dr 
i«l  x     x     j=l  J  °    J  °   t*  j=l  J   k=l  dyk   K 

s  T  s   r    of . 

=  S  z.  (t  )  6x  (t  )  +  J   E   S  z  -1  69  dr     (123) 

i=l  x  °    X  °   t0  i=l  j=l  i  38j   J 

which  is  the  same  as  Equation  (119)  and  equivalent  to  Equation  (15)  of 
Section  II.  Kiis  is  also  called  Green's  Identity. 

T/2ienever  the  variational  equation,  Equation  (116)  is  written,  a  formal 
solution,  Equation  (122),  can  be  obtained  by  using  Green's  functions. 
Green's  Identity,  Equation  (123),  follows  immediately  by  taking  the  inner 
product. 

Green's  functions  also  lead  to  computational  schemes  for  the  solution 
of  optimization  problems.  Solving  optimization  problems  by  means  of  the 
maximum  principle  results  in  two  point  boundary  value  problems.  Usually 
we  have  problems  in  which  the  initial  conditions  are  given  for  the  system 
equations  and  the  final  conditions  are  given  for  the  adjoint  system  of 
equations  (or  adjoint  systems).  An  iterative  technique  must  be  used  to 
treat  these  problems. 

Assume  that  8(t)  lies  in  the  6-neighborhood  of  9(t),  66 (t)  must  be 
assumed  to  be  approximately  zero.  Equation  (122),  then,  is  approximated  by 

6x,(T)=  2  G..(tn,t)  6x.(tn).  (124) 

1     j=l  1J  u     J  u 

Let  x'(T)  be  the  guessed  final  value  of  the  state  vector.  If  it  corresponds 

to  an  initial  value  x'(tQ),  instead  of  the  given  initial  value  x(tQ),  one 

can  obtain  a  better  guess  of  x(T)  by  using  the  following  iterative  equation 
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obtained  from  Equation  (12*0 

^(T)  =  x'(T)  +  E  0w(t0,  t)  [x.(tQ)  -  x^tQ)]  .         (125) 

3— 1 

This  scheme  is  based  on  the  work  of  Bliss  (26)  and  Goodman  and  Lance  (27). 
Dean  and  Aris  first  applied  it  to  the  problem  of  optimization  (#0.  Green's 
functions  can  also  be  associated  with  the  method  of  gradients  (or  method  of 
steepest  descent)  to  solve  the  optimization  problems  computationally  (23, 
28,  29). 

The  Weak  and  Strong  Forms  of  the  JSaximum  Principle 

Algorithms  have  been  derived  for  the  weak  form  of  the  maximum  principle 
for  both  the  simple  and  complex  discrete  (6)  as  well  as  continuous  systems 
by  using  the  first  order  variational  equations.  Mien  the  objective  function 
is  to  be  maximized  (or  minimized),  the  Hamiltonian  function  is  made 
stationary  with  respect  to  the  optimal  decisions  which  lie  in  the  interior 
of  an  admissible  region,  and  H  is  made  minimum  (or  maximum)  when  they  lie 
at  the  boundary  of  the  admissible  region. 

It  has  been  pointed  out  that  there  is  no  exact  analogue  of  the  discrete 
maximum  principle  to  the  continuous  one.  In  the  discrete  system  it  is  shown 
that  there  exists  only  the  weak  form  of  the  maximum  principle  (30,  31,  32,  33) 

The  algorithm  for  the  strong  maximum  principle  can  be  obtained  by 

considering  the  second  order  terms  in  the  variational  equations  (31,  32,  33). 

In  the  following  the  strong  form  of  the  maximum  principle  is  obtained  for 

simple  continuous  systams.  Taking  into  account  the  second  order  terms, 

Equation  (11)  of  Section  II  can  be  written  as 

dy.  _  _      2 

e  — i=  f.(x;  9)  -  f.(x;  6)  +  0(e  ) 

at    i       x 


% 


i  =  l.  2 s  (126)* 

v&ere  the  partial  derivatives  are  evaluated  along  the  optimal  trajectory 
and  0(e3)  represents  the  terms  of  order  e^  and  those  of  order  higher  than 
e?   such  that 

e  -*0   e" 

Using  Green's  functions,  one  obtains  a  solution  for  Equation  (126)  as 

t  Bfi 

ey.  (t)  =  S  G  (V  t)  ey.(t0)  +  J  Z    t^(t.r)  ^  ccpk  dx 


lim   ^£p  =  0. 


t   J.k.n  «     SxkS0m 


0 


♦§{*  ^BVV)^;KH^*^). 


"tn  -  t  -  T;     i  =  1,  2,...,  s.  (127) 

Successive  approximation  by  substituting  ey(t)  into  Equation  (127)  itself 
gives 


*  2  denotes  the  summation  over  the  j  and  k  subscripts,  j  and  k  range 
from  one  to  s  for  x  and  one  to  r  for  6. 
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2 

T  of 

tQ  j,k,m      "  km 


S2f, 


'0 
T 


Sf. 


+  -  f        E      G   .(T,t)  r rjj-  ecp    I  f      E    G.  .  r^  ecp    dt  J 

2  {      .,  ij  8x,  59„    Ym  Li     j  _    kj  o6m    Ym      J 

tn  J,k,m      J  ^c      m  tn  j,m      J       m 

ecp    dt 
Ym      J 

+  L{0J!kG^^e^dtJdT  +  0(e) 


dT 


tA  j.k, 


V>  5^'  & "*  5  "   dtUT 


(128) 


X   —    J-  j     £  j  ♦  ♦  *  )     S  ) 

Now  consider  the  special  variations  (30,  33) 


tQ  <  t  <  T. 


ecp(t)  =  < 


f  ecp(t)  ,     tx  <  t  <  tj  +  At 


(129) 


0   "  ,     otherwise. 
If  the  initial  state  is  assumed  to  be  fixed,  that  is 
ey(tQ)  =  0  , 

the  first  two  terms  on  the  right  hand  side  of  Equation  (128)  are  of  the 

2 

order  At  and  the  remaining  terms  are  of  the  order  (At)  and  of  order  higher 

than  (At)  .  Letting  At  be  sufficiently  small,  one  obtains 
t,+At  of, 

€7i(T)  =  I         j?k  Gi^(T'T)  ^k  ^k  dT 

,  t  +At 
+  ±  f  1  E   G,,(T,  t) 


52f, 


j.k.m 


13 


k   m 


1   ^V^m5  dT 


(130) 


+  0(e3)  +0[(At)2J  . 

2 

vjhere  0(At)  represents  second  and  higher  order  terms.  The   objective  function 

is  of  the  form 

S  ■  E  c.x.(T),        i  =  1,  2,...,  s 

i  x  1 
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where 

z±(T)  =  g±   ,        i  =  1,  2... 

>  • ,  s» 

As  in  Equation  (123),  taking  the  inner  product  of  Equation  (130)  and 

z(T) 

yields 

t-j+At         3f, 
2  ey  (T)  c .  =  f      2  z  (t)  rr^  «p  dr 
i   x        \    j,k  J    36k   k 

+■*/•       2   Z,(t) 

*  ^    j,k,m  J 

iL.  / 

Sek  S9m 

«?k>(«%)  dT 

+  0(e3)  +  0J_(At)2] 

-jfz*& 

„  1  ^1^  T    S2H 
2  ^    kTm  S9k  S9m 

(e*k)(% 

)  dT 

+  0(«3)  +  o[(At)2]  . 

(13D 

For  the  objective  function  to  be  a  maximum, 

,  one  has 

2  ey  (T)  c.  <  0  , 

i   1    x 

(132) 

The  weak  form  of  the  maximum  principle  requires  that 

-vo  =0»    k  =  l,  2,...,  r, 

t,  <  t 

<  t,+At. 

(133) 

Therefore, 

f  £^*vh>* 

<  0  . 

(13^) 

Since  t,  and  e^  are  arbitrary,  and  At  may  be  chosen 

as  small  as  one 

desires, 

the  Kessian  matrix  whose  kmth  element  is 
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2 

5H 

ae.  S9 

k   m 
mast  be  positive  definite  at  all  but  a  finite  number  of  points  (30,  31.  32). 

Hence  it  is  concluded  that  in  maximizing  (minimizing)  an  objective  function, 
the  Hamiltonian  function  must  be  a  maximum  (or  minimum)  at  all  but  a  finite 
number  of  points.  If  the  Hessian  matrix  is  semipositive  definite  one  can 
obtain  the  same  result  by  considering  higher  order  terms.  The  above  deri- 
vations using  tensor  and  matrix:  notation  were  obtained  by  Jackson  and  Horn 
(31,  32)  and  Denn  and  Aris  (33) • 

The  basic  difference  between  the  continuous  and  discrete  cases,  as 
Polak,  Jackson,  and  Horn  have  noticed,  is  that  in  the  former  one  can  use 
a  sufficiently  small  interval  of  time  At  to  simplify  the  result  while  in 
the  latter  no  small  interval  of  At  can  be  adjusted.  The  strong  maximum 
principle  for  different  cases  was  considered  by  Jackson,  Horn,  Denn  and 
Aris  (30,  31,  32).  So  far  as  applications  to  the  solution  of  problems 
are  concerned,  both  the  strong  and  weak  forms  of  the  maximum  principle 
lead  to  the  same  result  for  continuous  processes. 
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APPENDIX  I 

Pontryagin's  maximum  principle  (10)  can  be  summarized  in  the  folio-wing 
theorem: 

Let  6(t),  t  <  t  <  I  be  a  piecewise  continuous  vector  function  satisfy- 
ing the  constraints  given  by 

♦i  Lei^'  G2(t) 8r(t)j  <  °  '    3L"1,  2 m* 

In  order  that  the  objective  function  S  be  a  maximum  (or  minimum),  it  is 

necessary  that  there  exist  a  nonzero  continuous  vector  z(t)  satisfying 

fSw  SL  i  -  1  2     s  (1) 

dt^-SxT  *         '  


z.(T)  =  c,        i  =  1,  2 s  (2) 

and  that  the  vector  function  G(t)  be  chosen  that  H(z,x,G),  is  a  maximum 

(or  minimum)  for  every  t,  t  <  t  <  T.  Furthermore,  the  maximum  (or  minimim) 

value  of  H  is  constant  for  every  t. 

Based  on  this  theorem  it  can  be  shotjn  that 

max  (or  min)  H  =  0  , 
if  the  time  interval  is  not  fixed.  Assume  that  x(t)  and  9(t)  are  the 
optimal  trajectory  and  decision,  and  let  T  be  the  final  time  corresponding 
to  them.  Then  the  problem  may  now  be  considered  as  a  fixed  time  problem 
with  the  final  time  fixed  at  f  and  with  the  initial  given  conditions. 
Therefore,  the  objective  function  S  defined  for  a  fixed  time  problem  becomes 

S  =  2  c.x.(f).  (3) 

i=l  x  x 

and  correspondingly  the  components  of  the  adjoint  vector  take  the  value  of 
c  at  t  =  f ,  that  is 
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zi(T)  ■  c-  ,    i  =  1,  2 s.  (4) 

Consider  now  a  variation  of  the  objective  function  S  resulting  from 
a  very  small  change  of  time,  from  T  to  T  +  5T,  along  the  optimal  trajectory 
x(t).  One  has 


S3  =  S  c  x.  (T  +  6T)  -  c.  x.  (T) 


s 
=    S 

i=l 

dx, 
Ci  dt 

t=T 

s           _     dx. 

=  H 

6T  . 

t=T 

I 

6T 


+=r 


(5) 

Since  6S  must  be  greater  than  or  equal  to  zero  for  a  minimum  of  S  (or  6S 
must  bo  less  than  or  equal  to  zero  for  a  maximum  of  S)  and  6T  may  be  positive 
or  negative,  one  may  conclude  that 

6S  =  0  , 
that  is 


H 


=  0. 


t=T 


From  the  above  theorem  it  is  known  that  the  extremum  of  H  is  constant  for 
every  t,  tQ  <  t  <  T.  It  follows  that 
min  H  =  0   for   0  <  t  <  f . 


APPENDIX  II 

Consider  the  system  of  the  non-homogeneous  linear  differential  equations 

,n       ,n-l 

LW  -  p0    n*  +  *l  33T  +  .«♦  Vi  -  *!<*>•                (8) 

dt      dt 

.-  "   -L  |   »'-?  •  •  m   y   S  • 

u.(x)  =  y1  »    3  =  If  2 s-                     C9) 

There  exists  a  Green's  function  G^Ct.T)  such  that  (34) 

a) 

: 

the  solution  of  Equation -(8)  can  be  -written  as 

x.(t)  =  2  G..(t  ,t)  y,  +/   2  G  (t,T)  y,(t)  dr 

i    >±   ij  o   j  ^  ^  ij     i         (10) 

1  =  J.}  <~ }  •  •  •  f    s 

b) 

G.  .(t,r)  is  continuous  and  possesses  continuous  derivatives  of 

orders  up  to  and  including  (n-2)  -when  tg  <  t  <  T 

c) 

the  derivative  of  order  (n-1)  is  discontinuous  at  a  point  t  within 

(tQfT),  the  discontinuity  being  an  upward  jump  of  amount  1/pq(t) 

d) 

G.  .(t,T)  satisfies  the  differential  equation  at  all  points  of 

(tQ,T)  except  t. 

For  the 

variational  equations  of  the  performance  equations 

s  of .       r  of. 
fix.  =  2  r-i  fix.  +  2  r^  66,,   i  =  1,  2 s     (4.116) 

x   0=1  ^j   J   k=l  B9k   k 

by  using 

Green's  function  the  solution  may  be  written  as 

fix. 

s                 t  s         r  afi 
(t)  =  2  G..(t  ,t)  6x.(t  )  +  f   2  G  (t,r)  2  —^68  dr.   (11) 
j=l  XJ  u     J  u    tQ  k=l  1J     k=l  adk   k 

In  order 

to  relate  the  Green's  function  to  the  adjoint  vector  z(t),  one 
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defines  the  following: 

z.  .(T)  =  6.  .  , 

ij      10 

Gi;j(T,  t)  =  Zij(t),                                (12) 

Gi;j(t0,  T)  =  z..(t0).   . 

Since 

z±(T)  =  g±  ,                    i  =  1,  2,...,  s                  (2.6) 

or 

s          s 

z.(T)  =  c  =  2  c  6   =  2  c  z  (T).                (13) 
1      x   j=l  J  J1   j=l  J  31 

Thus  Equation  (11)  becomes 

s                T  s        r  of . 

ta(T)»  2  2  Ctj6x(t)  +  r   2  z     (t)     2  r^  60  dr.     (14) 

1       j=l   *J   °     J   °     lQ  j=l   3-J     k=l  9ek    k 

From  the  linearity  and  homogeneity  of  the  adjoint  system,  it  can  be  concluded 

that 

s 

z  (t)  =  2  c  z .  (t)     for    tn  <  t  <  T.            (15) 

1     j=l  x     J1             u  -   - 

The  inner  product  of  a  constant  vector  with  z.  .  gives  the  adjoint  vector,  z. 

t 
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NOMENCLATURE 

a  vector  belonging  to  a  tangent  plane 

b  number  of  initial  points 

c  number  of  continuous  branches 

c'  number  of  entering  streams  -which  are  continuous 

c"  number  of  leaving  streams  which  are  continuous 

c.  constant  in  objective  function 

d  number  of  discrete  branches 

d'  number  of  entering  streams  which  are  discrete 

dn  number  of  leaving  streams  which  are  discrete 

E  Weierstrass  excess  function 

f .  real  function 

F  real  function 

g  real  function 

G  Green's  function 

h  real  function 

H  Hamiltonian  function 

k  kth  branch 

m  mth  stage 

M  constant 

n  nth  stage 


N    total  number  of  branches  or  total  number  of  discrete  stages  or  the 
Nth  discrete  stage 

?    profit  function 

r    dimension  of  the  decision  vector 

R    region  of  the  space  of  real  variables 


s 

dimension  of  the  state  vector 

S 

objective  function 

t 

time                  » 

•*d 

initial  value  of  time  t 

t' 

time  as  a  dummy  variable 

T 

final  value  of  time 

W 

real  function 

X 

state  vector 

X 

optimal  state  vector 

.  * 

X 

derivative  of  x  with  respect  to  t 

y 

perturbation  of  the  state  vector 

z 

adjoint  vector  or  Green's  vector 

zij 

Green's  tensor 

Greek  Letters 

a 

constant 

P 

constant 

6ij 

Kronecker  delta 

e 

small  number 

n 

parameter 

Y 

constant 

8 

decision  vector 

e 

optimal  decision  vector 

5 

constant 

T 

parameter 

X 

constant 

t 

■     89 
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p,    constant 

9    perturbation  of  the  decision  vector 

$    real  function 
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The  main  purpose  of  this  work  is  to  develop  the  working  algorithms  for 
the  optimization  of  a  topologically  composite  processes  by  applying  the 
maximum  principle. 

The  basic  algorithms  of  the  maximum  principle  for  simple  continuous 
and  discrete  processes  are  first  presented.  A  composite  process  is  composed 
of  both  complex  continuous  and  discrete  processes.  The  algorithms  derived 
are  applicable  to  such  a  process  without  decomposing  the  process  into  simple 
subprocesses.  To  illustrate  the  use  of  the  algorithms,  two  examples  are 
solved. 

The  underlying  idea  of  the  maximum  principle  is  closely  related  to 
other  variational  techniques.  The  relationships  between  the  calculus  of 
variations  and  the  maximum  principle  are  discussed.  The  main  advantage  of 
the  latter  over  the  former  is  that  it  can  be  used  to  solve  problems  in  which 
the  decision  variables  are  constrained.  These  problems  can  not  be  solved 
using  the  calculus  of  variations.  For  the  unconstrained  case,  the  basic 
algorithms  of  the  maximum  principle  are  shown  to  be  equivalent  to  the  Euler- 
Lagrange  equation  and  they  conform  to  the  Weierstrass  necessary  condition. 
The  transformation  used  in  the  maximum  principle  is  essentially  the  same 
as  the  canonical  transformation  in  the  calculus  of  variations.  The  basic 
algorithms  of  the  maximum  principle  and  those  of  dynamic  programming  are 
compared.  Finally,  the  weak  and  strong  forms  of  the  maximum  principle, 
which  have  been  strongly  argued  recently  are  presented.  It  is  concluded 
that  there  is  no  exact  analog  of  the  continuous  maximum  principle  for  the 
discrete  maximum  principle.  The  discrete  maximum  principle  uses  only  a 
weak  condition,  that  is,  when  the  objective  function  has  a  maximum  (  or 
minimum)  the  Hamiltonian  function  attains  a  stationary  value.  The  continuous 
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maximum  principle  has  a  strong  condition,  that  is,  if  the  objective  function 
has  a  maximum  (or  minimum),  then  the  Hamiltonian  function  attains  its 
maximum  (or  minimum)  value. 
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